{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CPSC 330 Lecture 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lecture plan\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements (5 min)\n",
    "- Dataset of the week \n",
    "- Confusion matrices (5 min)\n",
    "- Precision, recall, F1 score (10 min)\n",
    "- Precision-recall curves (10 min)\n",
    "- Break (5 min)\n",
    "- ROC curves (10 min)\n",
    "- Addressing class imbalance (20 min)\n",
    "- Stratified splits (5 min)\n",
    "- CV with custom scores, hyperparameter optimization (5 min)\n",
    "- This week - summary (5 min)\n",
    "\n",
    "Piazza:\n",
    "\n",
    "- True/False questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Per the [poll on Piazza](https://piazza.com/class/kb2e6nwu3uj23?cid=269), I will start archiving/posting the Zoom chats, but to Canvas only (not publicly). \n",
    "  - This should happen automatically from now on. \n",
    "  - If you don't want your name archived on Canvas, you can change your name on Zoom. \n",
    "  - But if you don't care, I do like knowing who you are! \n",
    "  - Also, I didn't realize I could enable audio transcripts, so those are enabled gong forward. There should now be a \"CC\" button on the videos and I think they are also searchable. \n",
    "- hw4 posted, due Monday at 11:59pm\n",
    "  - A reminder that you need to form groups separately for each assignment on Canvas - please do that before the deadline!\n",
    "- The midterm will be on Thu Oct 22 in class\n",
    "  - 75-minute Canvas quiz in the 80-minute window\n",
    "  - Open book, no invigilation, questions in shuffled order\n",
    "  - Probably the \"you can't go back\" restriction, which is super annoying but makes cheating more difficult \n",
    "  - I'm thinking mostly short-answer questions\n",
    "  - Will cover up to Oct 15 lecture (will not cover Oct 20 lecture)\n",
    "  - Last year's (in-person) midterm posted at https://github.com/UBC-CS/cpsc330/tree/master/exams/2019W2\n",
    "  - I will post the solutions to last year's midterm on Canvas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Appropriate select between different classification scoring metrics (accuracy, precision, recall, F1 score, ROC AUC) given a problem.\n",
    "- Explain why accuracy is not always the best metric to optimize.\n",
    "- Interpret a confusion matrix. \n",
    "- Interpret an ROC curve and explain why a higher area under the curve is better.\n",
    "- Given a problem, determine when it makes sense to alter the probability threshold and/or use the `class_weight` hyperparameter in sklearn.\n",
    "- Explain how stratified splitting differs from standard splitting.\n",
    "- Use hyperparameter optimization tools in scikit-learn to optimize scores other than accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_validate\n",
    "from sklearn.preprocessing import normalize, scale, Normalizer, StandardScaler, OrdinalEncoder, OneHotEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import confusion_matrix, classification_report, roc_curve, roc_auc_score, plot_confusion_matrix, plot_precision_recall_curve, precision_score, recall_score, f1_score\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset of the week (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's classify fraudulent and non-fraudulent transactions using Kaggle's [Credit Card Fraud Detection](https://www.kaggle.com/mlg-ulb/creditcardfraud) data set.\n",
    "- For confidentially reasons, it only provides transformed features with PCA.\n",
    "  - So we don't need to worry about categorical features this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "cc_df = pd.read_csv('data/creditcard.csv', encoding='latin-1')\n",
    "df_train, df_test = train_test_split(cc_df, test_size=0.15, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(242085, 31)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_train.drop(columns=['Class'])\n",
    "y_train = df_train['Class']\n",
    "\n",
    "X_test = df_test.drop(columns=['Class'])\n",
    "y_test = df_test['Class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V20</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>113834</th>\n",
       "      <td>73222.0</td>\n",
       "      <td>-1.047911</td>\n",
       "      <td>0.264798</td>\n",
       "      <td>2.882793</td>\n",
       "      <td>1.050493</td>\n",
       "      <td>-1.754437</td>\n",
       "      <td>1.013074</td>\n",
       "      <td>-1.216518</td>\n",
       "      <td>0.993989</td>\n",
       "      <td>-0.289283</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.374125</td>\n",
       "      <td>-0.392708</td>\n",
       "      <td>-0.197803</td>\n",
       "      <td>-0.109727</td>\n",
       "      <td>0.396181</td>\n",
       "      <td>0.181980</td>\n",
       "      <td>-0.065889</td>\n",
       "      <td>0.328321</td>\n",
       "      <td>0.089068</td>\n",
       "      <td>12.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209720</th>\n",
       "      <td>137676.0</td>\n",
       "      <td>1.862475</td>\n",
       "      <td>-0.187619</td>\n",
       "      <td>-1.762891</td>\n",
       "      <td>0.300562</td>\n",
       "      <td>0.274679</td>\n",
       "      <td>-0.482736</td>\n",
       "      <td>0.026326</td>\n",
       "      <td>-0.034559</td>\n",
       "      <td>0.755427</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.035498</td>\n",
       "      <td>-0.157633</td>\n",
       "      <td>-0.394398</td>\n",
       "      <td>0.168381</td>\n",
       "      <td>0.589022</td>\n",
       "      <td>-0.118405</td>\n",
       "      <td>-0.144933</td>\n",
       "      <td>-0.023933</td>\n",
       "      <td>-0.014421</td>\n",
       "      <td>69.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6086</th>\n",
       "      <td>6942.0</td>\n",
       "      <td>1.475676</td>\n",
       "      <td>-0.082584</td>\n",
       "      <td>-0.176180</td>\n",
       "      <td>-0.551063</td>\n",
       "      <td>-0.255257</td>\n",
       "      <td>-1.112522</td>\n",
       "      <td>-0.028757</td>\n",
       "      <td>-0.549704</td>\n",
       "      <td>-0.061398</td>\n",
       "      <td>...</td>\n",
       "      <td>0.128606</td>\n",
       "      <td>0.033541</td>\n",
       "      <td>0.348765</td>\n",
       "      <td>-0.215062</td>\n",
       "      <td>0.059112</td>\n",
       "      <td>0.866292</td>\n",
       "      <td>-0.106655</td>\n",
       "      <td>-0.046937</td>\n",
       "      <td>-0.004705</td>\n",
       "      <td>15.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15352</th>\n",
       "      <td>26742.0</td>\n",
       "      <td>0.028552</td>\n",
       "      <td>-2.235673</td>\n",
       "      <td>0.517263</td>\n",
       "      <td>0.642202</td>\n",
       "      <td>-1.330937</td>\n",
       "      <td>1.093816</td>\n",
       "      <td>-0.222273</td>\n",
       "      <td>0.376028</td>\n",
       "      <td>0.885882</td>\n",
       "      <td>...</td>\n",
       "      <td>0.932355</td>\n",
       "      <td>0.196759</td>\n",
       "      <td>-0.399784</td>\n",
       "      <td>-0.342213</td>\n",
       "      <td>-0.203934</td>\n",
       "      <td>-0.255216</td>\n",
       "      <td>0.896698</td>\n",
       "      <td>-0.107922</td>\n",
       "      <td>0.091284</td>\n",
       "      <td>549.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201339</th>\n",
       "      <td>133843.0</td>\n",
       "      <td>1.964985</td>\n",
       "      <td>-0.418204</td>\n",
       "      <td>-0.380086</td>\n",
       "      <td>0.428730</td>\n",
       "      <td>-0.753728</td>\n",
       "      <td>-0.668689</td>\n",
       "      <td>-0.557784</td>\n",
       "      <td>-0.048365</td>\n",
       "      <td>1.359894</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.238540</td>\n",
       "      <td>0.216086</td>\n",
       "      <td>0.751400</td>\n",
       "      <td>0.136596</td>\n",
       "      <td>-0.086692</td>\n",
       "      <td>-0.173467</td>\n",
       "      <td>-0.235863</td>\n",
       "      <td>0.030625</td>\n",
       "      <td>-0.036667</td>\n",
       "      <td>19.78</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Time        V1        V2        V3        V4        V5        V6  \\\n",
       "113834   73222.0 -1.047911  0.264798  2.882793  1.050493 -1.754437  1.013074   \n",
       "209720  137676.0  1.862475 -0.187619 -1.762891  0.300562  0.274679 -0.482736   \n",
       "6086      6942.0  1.475676 -0.082584 -0.176180 -0.551063 -0.255257 -1.112522   \n",
       "15352    26742.0  0.028552 -2.235673  0.517263  0.642202 -1.330937  1.093816   \n",
       "201339  133843.0  1.964985 -0.418204 -0.380086  0.428730 -0.753728 -0.668689   \n",
       "\n",
       "              V7        V8        V9  ...       V20       V21       V22  \\\n",
       "113834 -1.216518  0.993989 -0.289283  ... -0.374125 -0.392708 -0.197803   \n",
       "209720  0.026326 -0.034559  0.755427  ... -0.035498 -0.157633 -0.394398   \n",
       "6086   -0.028757 -0.549704 -0.061398  ...  0.128606  0.033541  0.348765   \n",
       "15352  -0.222273  0.376028  0.885882  ...  0.932355  0.196759 -0.399784   \n",
       "201339 -0.557784 -0.048365  1.359894  ... -0.238540  0.216086  0.751400   \n",
       "\n",
       "             V23       V24       V25       V26       V27       V28  Amount  \n",
       "113834 -0.109727  0.396181  0.181980 -0.065889  0.328321  0.089068   12.00  \n",
       "209720  0.168381  0.589022 -0.118405 -0.144933 -0.023933 -0.014421   69.32  \n",
       "6086   -0.215062  0.059112  0.866292 -0.106655 -0.046937 -0.004705   15.00  \n",
       "15352  -0.342213 -0.203934 -0.255216  0.896698 -0.107922  0.091284  549.81  \n",
       "201339  0.136596 -0.086692 -0.173467 -0.235863  0.030625 -0.036667   19.78  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "113834    0\n",
       "209720    0\n",
       "6086      0\n",
       "15352     0\n",
       "201339    0\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- At this point we would want to think more carefully about the Time and Amount variables.\n",
    "- We could scale Amount, but do we want to scale Time?\n",
    "- More on time series coming later in the course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Confusion matrices (5 min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.032059\n",
       "score_time     0.002829\n",
       "test_score     0.998294\n",
       "train_score    0.998294\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy = DummyClassifier(strategy='prior')\n",
    "pd.DataFrame(cross_validate(dummy, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holy cow! `DummyClassifier` is getting 99.8% accuracy!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    241672\n",
       "1       413\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We have MANY non-fraud transactions and only a tiny handful of fraud transactions.\n",
    "- So, what is a good accuracy here? 99.9%? 99.99%?\n",
    "- Or, maybe accuracy is not a meaningful measure anymore.\n",
    "- Today's class is all about this issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr = Pipeline([\n",
    "    ('scale', StandardScaler()),\n",
    "    ('lr', LogisticRegression(max_iter=1000))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       1.435038\n",
       "score_time     0.019517\n",
       "test_score     0.999219\n",
       "train_score    0.999220\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cross_validate(pipe_lr, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This seems slightly better than `DummyClassifier`.\n",
    "- But can it really identify fraudulent transactions?\n",
    "- Today we're going to cover some new tools.\n",
    "- These are a bit easier to deal with if we have an explicit validation set instead of using cross-validation.\n",
    "- Since our dataset is quite large, we can do this without worrying too much.\n",
    "  - Let's do an 80-20 split. \n",
    "  - We can do this with `train_test_split` again, or just manually since the original `train_test_split` already shuffled the order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_fold = X_train[len(X_train)//5:]\n",
    "y_train_fold = y_train[len(y_train)//5:]\n",
    "\n",
    "X_valid_fold = X_train[:len(X_train)//5]\n",
    "y_valid_fold = y_train[:len(y_train)//5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_fold, X_valid_fold, y_train_fold, y_valid_fold = train_test_split(X_train, y_train, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note my naming convention: `X_train` is everything except `X_test`, whereas `X_train_fold` is a subset of `X_train`.\n",
    "- The first thing we're going to introduce is the _confusion matrix_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr.fit(X_train_fold, y_train_fold);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbEAAAEQCAYAAAA+kEr6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1jElEQVR4nO3dd5hU5dnH8e+9hd47KAgKYkVMLCGvYokRE3uJsYtGjUZ9E18laoCI2DXGkqjBGMWIijWJLUqMghWjUVFRsdB7WZa2sGy53z+eszA7zO7OLmdm2d3f57rmmp1z7jnzzBmYe556zN0RERFpiHLquwAiIiJ1pSQmIiINlpKYiIg0WEpiIiLSYCmJiYhIg5VX3wWQrdelU6737Z1f38WQWvjqk1b1XQSppTWsXO7uXev6/GGHtPYVBWVpxf73k+JX3P2Iur5WU6Ik1gj07Z3Pf17pXd/FkFoY1mtwfRdBaulVf3rO1jx/RUEZ/3mlT1qxuT2/7rI1r9WUKImJiGSBA+WU13cxGh0lMRGRLHCcEk+vOVHSpyQmIpIlqonFT0lMRCQLHKdMy/zFTklMRCRLylESi5uSmIhIFjhQpiQWOyUxEZEsUU0sfkpiIiJZ4ECJ+sRipyQmIpIFjqs5MQOUxEREssGhTDksdkpiIiJZEFbskLgpiYmIZIVRhtV3IRodJTERkSwIAzuUxOKmJCYikgVhnpiSWNyUxEREsqRcNbHYKYmJiGSBamKZoSQmIpIFjlFGTn0Xo9FREhMRyRI1J8ZPSUxEJAscY6Pn1ncxGh0lMRGRLAiTndWcGDclMRGRLNHAjvgpiYmIZIG7UeaqicVNSUxEJEvKVROLnZKYiEgWhIEd+sqNm86oiEgWaGBHZiiJiYhkSZnmicVOSUxEJAu0YkdmKImJiGRJuUYnxk5JTEQkC8ICwEpicVMSExHJAsco0bJTsVMSExHJAnc02TkDlMRERLLCNNk5A5TERESywFFNLBOUxEREskQDO+KnJCYikgWO6aKYGaAkJiKSBQ6UaO3E2OmMiohkhel6YhmgJCYikgWOVuzIBCUxEZEsUU0sfkpiIiJZ4G6qiWWAkpiISBaEgR1adipuSmIiIllhmuycATqjIiJZEAZ2WFq3dJjZ9mb2BzN718yKzMzNrG9STN9oe6pbh6TYFmZ2m5ktMrP10XGHpnjdHDO72sxmm9kGM5tmZidWUcbzzexLMys2sxlmdmEVcceZ2UfR8eaY2SgzS6vaqiQmIpIlZeSkdUtTf+BkYCXwZg2xNwFDkm5rkmL+ApwP/BY4ClgEvGJmg5PirgPGAH8EfgRMBZ4ysx8nBpnZ+cA44BngCOAp4F4zuygpblgU8350vLuAUcCNNbwnQM2JIiJZkYEVO95w9+4AZnYecHg1sTPdfWpVO81sL+A04Fx3fyjaNgWYDowFjom2dQOuAG52999FT3/dzPoDNwMvRXF5wA3AI+4+MiGuF3CdmT3g7iXR9puBt9z9goS4NsAoM7vD3RdXdxJUExMRyZJyctK6pcPdy2Ms2jFACfBEwvFLgYnAMDNrHm0eBjQDJiQ9fwKwp5n1ix4PAbqmiHsE6AwcAGBmvYHBVcTlE2pm1VJNTNK2bGE+T97Tja8/acXM6S0p3pDDw+99To/eGyvFfftZS/5yY0+m/6c1OTkwaMhaLhizgO36bY5bMj+fe0dtz8zpLSlckUeLluX03WUDJ1+8hH0P3dzKUbQ2h99f3ptvPm1FwZI88vJh+502cOy5y/nBiSsrve6/nuzIu5Pa8/W0Vixd0IwfnlzAFXfOTfleitcbT97Tndf+1pFlC/Jp3a6MnQcX8dsHZpPfzGM8a43HAUcWcshxhQzYq4gOnUtZurAZb7/Unol3d2P9utB90X/PIs65ajF9d1lPu45lrF2dyzeftuSxO7vzxX9b1/M7qF/uUFJeb/WGm8zsT8A6YAow0t0/Tdi/OzDL3YuSnjedkLT6R3/vDhQD36SIA9gNmBXFAXxWTdzrVcW5+ywzK4riqqUkJmlbOLs5bzzfgQGD1rPH/mv575R2W8QsmNmMy4/vT99dNnDlH+dQVmY8+vseXHHCAO771ww6dCkFYP26XNp3KuXsKxfRpWcJRWty+OdjnRl1xk6MfmAWB/x4FQClG43cXDjlkiV0772Rko3GlOc6cuulO7BqRR4nXLBs02v/+5lOrFqRx3eGruGNFzpU+T5KS2DUGTuxeG4zfnrpEnbYeQOFK/L48I22lJfFe84ak5MuXMayBfk8dFNPli/Kp/8e6znj8sXs9f21XHZMf9yNNu3LWDi7GZOe7EjBknw6dCnlhAuWcdsz33L5cf2Z8XGr+n4b9SY0J6adxLqY2QcJj+939/vr8LLFhH6pScAyYBfgN8A7Zrafu38RxXUi9K0lK0jYX3Ff6O7Jv/RSxZHimOnGVWzrlGJ7JVlNYmY2HHgIWAX0c/eVCfvyCNXZa919TJbLNQS4E9gDaAXs7e4fZ7MMqZiZUw/noyp7fm8tT3wSfkj989FOKZPYk/d0JycXrp8wkzbtQ0bYZe8izvmfXXn6vq6cN3oRAH0HbuD/fj+v0nP3P2w1Z+2/G5MmdtqUxNp1KuPqe+dUitvvB2tYMLM5r0zsVCmJ3fj4t+RE3xEfTG5b5ft45k/d+ObTlox7/Uu6bVeyafuBR65K91Q0Sdec3Y9VBZu/Mj6d2oY1hbmMuHseg76/lmlvt+Xjt8It0Qevt+XJz6bzg5MKmnQSg1qt2LHc3ffZ2tdz90VA4ojAN83sZUKNaCRwRrTdCAMokyUXuDZxVBGbblxaJ6u+6rbtgSvr6bVT+QshoR9NaMv9qn6Ls23KSeNfyxcftmLX767blMAAuvYqoe/ADbz9codqn5ubB63blZGXX3NzXtuOpeTlVY5Lp3wAzz/chQOPKqyUwKRmiQmsQkVS6tKj6nO5oSiHko1GaUnTXnIp7iH2dS6H+zzgLWDfhM0FpK71dEzYX3Hf0cySC5kqjhTH7JRmHECHhP1Vqq8kNgm41Mx61NPrb2JmOcBA4EV3f83dp6ZoF66IbZ5qu2yWk0vKJJTfrJxFs5uxcUPlf/vl5VBWCgVL83j0ju4smNmco4cv3+L57iFudUEuL03ozH8nt+O485dtEVeTpfPzWbawGT122MgdV/Tm+J335Kh+g7jy5J349rOWtT5eUzdoyFoA5n7dotJ2Myc3z+m63UYuvmEBAC8/1jnr5du2hObEdG5ZKUzl2s90oJ+ZJVeVdwM2srkPbDrQHNgpRRzA5wlxsLnPq1Zx0Xy3VglxVaqvJHZ9dD+y2ijAzPYzs1fNbK2ZrTOzf5vZfkkx481svpntbWZvRhP/vq5qYl3C84YDZYTzMDqaADg76ZhDzOwdM1sP3BrtO8XMXjOzZVG5PjKzs5OOXTHJcHjS9oOj7QcnbMs1s+ujSYZFZjbZzJI//AZh+5028M2nLSlN+GFetDaHOV+1xN1Ys6ry/MUHruvFj/sM5tTBe/DUvd24+r457H3g2i2O+9xDXfhxn8H8ZI89uWfkdlw0dj4//EmqZvTqrViSD8BT93Rj8dxmXH3fbK6+dw6rVuQx4qT+LJ2fX+tjNlWde5Rw1oglfPhGG77+pPJ338hxc3hp7idMeP8LDjhyFaPP7LdFomuKyrG0bplkZn2A/wHeS9j8HGE04E8S4vKAnwKT3L042vwyIamdnnTYM4DP3H1W9PhdYHkVcQXA2wDuPheYVkVcCfDPmt5PfQ3sWESYKPcrM/udu89JFWRmgwgjaT4HhhN+OVwFTDGz77n7tITwdsBjhL6tscA5wH1mNsPdX6+iHC8Shnq+RWhSfIDQEVqhPWGI6e8InaHro+07Ak8T5jeUA0OBB8yspbv/Kc1zkGhMdPzfE2qp+xD+UTU4x/9sOW8+35G7r+zNWSMWU1YG91+7HevXhd9LyU1+x5+/jIOPW0nB0nxefbojN1+8A6Pun833fri6UtxBx65k1++uY1VBHlMntefeUduTkwtHnrmiVuUrjwYlN29ZzrXjZ9KiVfgxOmBQEef+z648N74L541aVLc334S0aFXGmIdmUVYKt1/We4v9D1zfkyfv6UbXXhs5evgKxj48i6t+utMWya4pCaMT41070cxOiv78bnT/IzNbBixz9ylmdjvhR/q7hIEdA4GrCd9bmyYTu/vHZvYEcKeZ5RNGGF4E9CMhwbj7UjO7A7jazNYAHxIS3aHAsQlxJWY2mjC5eQHwahRzLnCpuycOaf4N8IKZjQMeB/YmTHa+q6Y5YlC/oxNvAX4OXEN4Y6n8lpBUfuDuhQBm9i9gdvS8ExJi2wK/qEhYZvYGYfLfqYShnFtw92VmVvFzfn6KyYBtgDPc/R9Jz9v04UfNkZOBnoQPvVZJzMw6ApcRRh9dEW2eZGZlhCRZ1fMuAC4A6LPdtjPIdPf91nHJjfN48KZevDIxNB8NPmANP/xJAa8925G2HUorxXftVULXXiXAer73w9WMOLE/fx7ba4sk1qFzGR06h98Q+x6yhuL1Ofx5bC+GnbKCvFpUntp1DH11u+27blMCA+i2XQnb9y9Wk2Ia8puXc+342fTss5ErTtyJ5YuabRGzeG5zFs+Fr6a14r1X2zHutRkMv3IxI0/fsR5KvG3IwGRnCKtgJLo3up8CHExorruIUAloS6gdvUYYMDYj6bnnECYoX0/oj5oGHOHuHybFjQTWAr8EegAzgJPd/fnEIHf/UzQ47XJgBDAXuMTd702KeylKxtdE5VxCSLA3pPH+6y+JuXtB9CvhGjO7Bfg2RdhQ4IWKBBY9b7WZPUcYhJGoKLHG5e7FZvY10GcrilkKvJC80cwGEGp7QwkfYkX9ojg5Ng17Aq2BJ5O2T6SaJBYNt70fYJ+9WmxTE5uOHr6CYacWsHB2c1q1KaPbdiWMPH1HBu5dVGPCGTCoiL890LXG1xgwqIh/PdmJlcvyoySYnp47FNO8RTlbdEsDOJim/1crN88Z/efZDNy7iKt+uiOzv6w56ZeW5DDri5bstPv6GmMbu7ibCt2rz4ru/iDwYJrHWg/8X3SrLq6MkOiury4uih1HGOJfU9yzwLPplDNZff+XvYPQPjq2iv2dCE2PyRazeTRMhVQdJMXA1jTEL40+sE2i5VD+BexFaNo8kDDK50FCh2dt9YzulyRtT37coDRr7vQduIFu25Uw64sWfPRmW446a8sBG4nKy2H6+63ptUPNvwU+ndqGlq3LNs07S1dePux32Go+e681G4o2//NfOj+f+d82Z+DglGN6hDBY46o/zmHvA9Yy5py+fPlhepOXm7csZ8BeRSycs2WNrSnZVkYnNjb12g7l7mvN7CbgduC2FCEFhJpOsh6kMfQyBqlqOEOAHYAD3f2tio1RJ2iiDdF98v/c5CFaFUm6O5tH6lQ83ua8+UJ7AL7+NPRtvP9aWzp0LqV951IGDVnHsoX5vPDXzuy2TxH5zcr5+pNWTPxDd/7nR4UccnzhpuM88rserCnMZbd919GpWwkFS/N55fFOzPioFVfds7mL9MVHOvPFf1vxnaFr6dJzI6tX5vHGcx1484UOnPubhZVW15jzVXPmfhV+sxRvyGHJ/PxN5d1zyFo6dA6/R868YhH/++OdGX3mjpz486VsLDYm/L4HrduVccw51SfapuySGxcw9JhVPHZnNzYU5bDLd9Zt2rd8UT7LFzXjf2+Zx5rCPL6e1pJVBXl0334jx5yznE7dSrnt0q1pFGkcdFHM+G0LnSn3EqqvqaqmU4Ajzaytu68BMLO2hKbEyVkrYWUVPdOb2rCifq1jk+KWEGqCeyRtPzLp8SeEpWBOJrRVVzhlq0uaAddf0K/S4z9eHTr1Bw1Zy23PfENevvPlh6156ZEurF+XQ88dNnL6ZYs5/rzKw+H77xmaDSf/owNFa3Lp2LWUHXdbz+1/+4bd99v85dh3l/W8+0p7/jy2F2sKc2nXqZTe/YsZ+9eZ7H9Y5X6zN57ryITfb/7N88k7bfnknTDx9tanv6HD98Ooxx12LuaWp77lLzf05MYL+5Kb7+z1/bVc8+AsOnatXc2uKdnn0HC+T/vVUk771dJK+x65vTsTbu/Blx+14kenFfDj01fQolU5yxfnM+OjVvz+8t5pNT02Zu5GqZJY7Oo9iUV9V2OJ+neSXEe4JMC/o34zJ0ySbkXVTZCZ9g6wGrjHzK4h9GeNInSYtq8IcnePRvv8zMy+InR+HknobCUhrjAa7TMyGu0zidA8+bMsvJdae2Xhx9Xu79i1lFueTNW9WdmQYasZMmx1jXG771vE9RNmplW2M69YzJlX1DiYCQiriNz2dM3llM3O3r/GZeyYNLEzkyY29flgVVNTYfy2lZ8FDwFfJ290908IX/qrgYcJKxuvBQ5KGl6fNe6+DDgeyCUMs7+JMDQ/eRVmCKN3niUMoX+C0D93aYq4MYTROGcShtYfzpYDV0SkAVOfWGZktSbm7uOB8Sm2lwI7V/Gc94DDajju8Cq2H5xGmUpJsUZXVceM9r1GmMuQbExSXCEhMSWzpLgyQm1uVHVxItKwKUHFr96bE0VEmoIMzRNr8pTERESyJNNLSjVFSmIiIlngDqX1d1HMRktJTEQkS9ScGD8lMRGRLFCfWGYoiYmIZEkNSx1KHSiJiYhkiQZ2xE9JTEQkC9zVJ5YJSmIiIllhlGl0YuyUxEREskR9YvGrMomZWTmpL0WSiru7EqKISBUq1k6UeFWXeMaSfhITEZHqeOgXk3hVmcTcfUwWyyEi0uhpdGL8at0EaGZtCFcnXujuJTXFi4hImOysgR3xS/uMmtlRZvYhsAqYCewZbX/AzE7LUPlERBoN9/Rukr60kpiZHQf8g3D14iupfJ2rWcDZsZdMRKSRcbe0bpK+dGti1wAPufvhwJ1J+z4D9oizUCIijU2oZSmJxS3dPrFdgV9HfydXdlcS+shERKQaGmIfv3ST2GqgSxX7+gLLYimNiEgjpv6u+KXbnPgv4Goz65Cwzc2sOXAJ8M+4CyYi0pg4Rnl5Tlo3SV+6NbGRwH+AGcBLhCbFq4BBQHvguEwUTkSkMVFFLH5ppXx3nw18B3gB+CFQBgwFpgL7u/vCTBVQRKRR0MCOjEh7srO7zwd+lsGyiIg0bqqKxa4uK3b0ArYDFqgGJiKSPtWy4lebFTvOMrNZwDxCM+I8M5tlZmdkrHQiIo2EA+XlltZN0pdWTczMLgHuBl4FrgOWAN2BU4GHzay9u9+TsVKKiDR0DqgmFrt0mxMvB8a7+7lJ2x80s/HAFYCSmIhINTRPLH7pNif2ACZWse8xQq1MRESq42neJG3p1sQ+BXaqYt8AwvqJIiJSJQ2fz4R0k9gvgYlmthx41t3LzCwXOBEYAZySqQKKiDQaqmXFrsokZmbzqHzK2xOaFMvMbCXQEcgF1gJPADtksJwiIg2bg2vkYeyqq4n9G/1uEBGJkZJY3KpMYu4+PIvlEBFp/FQtiF2tV+wQEZE6UhKLXa2SmJntBQwEWiTvc/e/xlUoEZFGR5OdMyLdFTs6AC8C36vYFN0n/q5QEhMRqYYmO8cv3cnONwKdCZdfMeB44FDgUWAmsF9GSici0piUW3o3SVu6SWwYIZFNjR7Pd/fJ7n4WYT3FX2aicCIijYl5ere0jmW2vZn9wczeNbMiM3Mz65sirqOZPWBmy81snZm9amZ7pohrYWa3mdkiM1sfHXdoirgcM7vazGab2QYzm2ZmJ1ZRxvPN7EszKzazGWZ2YRVxx5nZR9Hx5pjZqGguco3STWI9gZnuXgZsANom7HsWODLN44iINE3pLjmVfpNjf+BkYCXwZqoAMzPgOeAI4FLCAhX5wOtmtn1S+F+A84HfAkcBi4BXzGxwUtx1wBjgj8CPCJWbp8zsx0mvfT4wDngmev2ngHvN7KKkuGFRzPvR8e4CRhEqTjVKd2DHYqBD9PccYAgwOXrcP81jiIg0YRb3wI433L07gJmdBxyeIuYY4ADgUHd/PYp9F5gF/Br432jbXsBpwLnu/lC0bQowHRgbHQcz60ZY8P1md/9d9Bqvm1l/4GbgpSguD7gBeMTdRybE9QKuM7MH3L0k2n4z8Ja7X5AQ1wYYZWZ3uPvi6k5CujWxtwiJC+AR4BozG2dm9wC3Aa+keRwRkaYrxpqYu5enEXYMsLAigUXPWwU8DxybFFdCWH2pIq6UsErTMDNrHm0eBjQDJiS9zgRgTzPrFz0eAnRNEfcIYXzFAQBm1hsYXEVcPqFmVq10k9i1wMvR37cRLrtyJOF6Ys8RqqkiIlKd8jRv8dmd1Au0Twf6RDWeirhZ7l6UIq4Zm1vcdgeKgW9SxAHslhBHitdOK87dZwFFCXFVSqs50d2/Bb6N/i4hXF/s8nSeKyIi1HaeWBcz+yDh8f3ufn8dXrUTMDvF9oLoviNh/dtOhL61quI6JdwXum8xWSBVHCmOmW5cxbZOKbZXohU7RESyJN2Rh8Byd98njpckdQNlcjbNRBxVxKYbl1bGr24V+9+mc4CKArj7dbWIFxFperI/2bmA1LWZjtH9yoS4PtXEFSTcdzQzS6qNpYojeu1FCXGdqolL1iFhf5Wqq4mNqenJCZww7FJERLYd00k9anE3YK67r02IO97MWiX1i+0GbGRzH9h0oDnhIsnfJMUBfJ4QB6HPa1Gace9WBEXz3VolxFWpulXs0x30IfXsq09aMazX4PouhojUoBbNiXF5DjjHzA5y9ykAZtYOOBp4LCnuWuAnwMNRXB7wU2CSuxdHcS8TktrpUXyFM4DPogEZEBLS8iju1aS4AuBtAHefa2bTorgHkuJKgH/W9AbVJyYikg1O7EtKmdlJ0Z/fje5/ZGbLgGVR0nqOkFAmmNkIQvPh1YT+pls3Fc39YzN7ArjTzPIJ88guAvoREkxF3FIzuwO42szWAB8SEt2hJAzZd/cSMxtNmNy8gJDIDgXOBS51940Jb+M3wAtmNg54HNibMNn5rprmiIGSmIhI9sRfE3sq6fG90f0U4GB3Lzezo4DfRftaEJLaIe4+L+m55xAmKF9P6I+aBhzh7h8mxY0kjGj8JdADmAGc7O7PJwa5+5/MzAkj2UcAc4FL3P3epLiXomR8DTAcWEJYreOGdE6AkpiISJbE3ZzoXvOYfXcvINSAzq0hbj3wf9GturgyQqK7Po3XHkdYeqqmuGcJSxjWmpKYiEi26FIssVMSExHJFiWx2CmJiYhkQW0usyLpq1USM7NBhAtjdgbGufviaPXiJe6+JhMFFBFpNHTBy9illcSiFYwnACewedmR5wmXaLkV+Aq4KkNlFBFpFFQTi1+6E5pvAA4DzgS6U3lNq38SlucXEZHqxHtRTCH95sRTgVHu/liKS0bPAvrGWioRkcZGfWIZkW4S6wx8UcW+HMJaWiIiUh0lsdil25w4i81Xdk62H2HGtoiIVMPK07tJ+tJNYn8FrjKz0wlX+QRwMzsEuAx4MBOFExERqU66SexW4EXgETZf3+UtwqKOL7v7HzJQNhGRxkUDO2KXVp9YtFbWKWZ2D2EkYjdgBSGBTclg+UREGgcN7MiIWk12dvc3gTczVBYRkcZNSSx2WnZKRCRblMRil+6KHeXUcPrdPXn+mIiIRAyNPMyEdGtiY9kyiXUGDifMERsfY5lERBof9YllRLoDO8ak2h6t3vE8sCrGMomINE5KYrFLd4h9StGoxXuBX8VSGhGRxkxD7GMXx8CO5kCnGI4jItKoqTkxfukO7OiTYnMzYA/gZuCDOAslItIoKYnFLt2a2GxSn34DvgUujqtAIiKNkmt0Yiakm8TOSbFtAzAHeD/qGxMRkeqoJha7GpNYNALxY2Chuy/LeIlERBop9YnFL53RiU7o89o7w2UREWncNDoxdjXWxNy93MzmAa2zUB4RkcZJCSoj0p0nNg74lZk1qzFSRES2YITmxHRukr50B3a0BXYCZprZy8AiKv+mcHe/Ju7CiYg0JkpQ8asyiZnZTOB4d58G/CZh17kpwh1QEhMRqY6SWOyqq4n1JazGgbtv1fJUIiKCklgG6HpiIiLZoP6ujKgpiemUi4jERd+osaspiV1rZsvTOI67+9lxFEhEpLHSslPxqymJDQaK0ziOfl+IiNRAzYnxqymJHefu/8lKSUREGjNNds4IDewQEckWJbHYKYmJiGRBxYodEi8lMRGRLLFyZbG4VZnENMFZRCRG6hPLCNXERESyRM2J8VMSExHJFiWx2CmJiYhkiWpi8VMSExHJFiWx2CmJiYhkg2vZqUzQCEQRkSyI+8rOZnawmXmKW2FSXEcze8DMlpvZOjN71cz2THG8FmZ2m5ktMrP1ZvaumQ1NEZdjZleb2Wwz22Bm08zsxCrKeL6ZfWlmxWY2w8wuTO/dpU81McmKLj03cvLFSxkwaD077r6eFi2ds/bblSXzm22Kadm6jDMuX8KAQUX033M9rduWM+LEnfjk3TZbHK9772LOH72IvQ9cQ14+zPioJX++rhdff9Iqm2+r0TrgyEIOOa6QAXsV0aFzKUsXNuPtl9oz8e5urF+XC9Tu88pvXs7Zv17MoSespE27Mr6d3pK/3NCTz97bMrZR84y0J/4v8H7C49KKP8zMgOeAfsClwErgauB1Mxvs7vMTnvcX4EhgBDATuBh4xcyGuPvHCXHXAVcAI4H/AqcAT5nZUe7+UsJrnw+MA24CXgV+ANxrZubu98XxxqGJ18TMbHgVv2TczA7bBso32cwm13c54tCr70aGHr2Ktaty+ey91ilj2nUsY9hPCygvMz56o22Vx2rbsZTf//0b+u6ygbuv3J4bL+oDwK1Pf0vv/hsyUv6m5qQLl1FeBg/d1JORp+/Iiw935qizlnPTxJlYVFVI9/MC+L/b5/Gj01bwyG09+O3Z/ShYms+Nj81kx93XZ+PtbDPirIkl+MLdpybcPkjYdwxwAHCmuz/u7i9H23KAX28ql9lewGnAZe7+Z3f/N3AyMBcYmxDXjZDAbnb337n76+7+c+B14OaEuDzgBuARdx8ZxY0CxgPXmVl+rd9lFVQTC34CzE/a9nl9FKSx+nRqa07Za3cAjjhtBfscvHaLmCXz8zlp9z0A2PvANRxw5KqUxzrqrBV07FrKiBP7s3B2cwA+fqsND0/9krOuWMwNF/bNzJtoQq45ux+rCjZ/PXw6tQ1rCnMZcfc8Bn1/LdPebpv257Xjbus59IRCbr+sN5Oe6ATAJ++24f7JMzhrxGLGDO+X+Te0Laifyc7HAAvd/fVNxXBfZWbPA8cSanEVcSXAEwlxpWY2EbjKzJq7ezEwDGgGTEh6nQnAg2bWz91nAUOAriniHgHOISTW14lBk66JJfg46ZfMVHdfnRxkZs3ro3CNgbulEZVODOz6nXUsmNV8UwIDKF4fanj7/XA1ObkaAra1EhNYhRkfh6baLj1Koi3pfV7fO3w1JRuNKc912LStvMyY8o8OfPegNeQ3azqjHaw8vVstPWpmZWa2wsweM7M+Cft2Bz5L8ZzpQB8za5MQN8vdi1LENQP6J8QVA9+kiAPYLSGOFK+dHLfVlMSqkNDUONTMnoo6S9+L9u1rZk+b2fyoA3SGmd1oZi2TjjHbzManOLab2ZikbackdIBON7PjM/fuGrbycqN045ZfoCUbjRYtnV5907kEntTWoCGh9jz36xa1et4OAzeweF4zitdX/rqZM6MFzZo7vfpujK2M27qYk9gq4HbgPOBQQl/VYcC7UbMfQCdCP1iygui+Y5pxnRLuC9236NxLFUeKYybHbTU1Jwa5URtuhcQP6FHgceAkNp+vPsDHhPbdNYRfHb8FdiR0ctZK1P/2GPAicDmhGn4XkA/MqO3xGrt53zZn76FraNuxlDUrw0di5gwcHH5Etu1QVp/Fa5Q69yjhrBFL+PCNNrUePNO2QylrC3O32L4m2ta2Q+kW+xolpzYDO7qYWWLf1v3ufn+lw7l/BHyUsGmKmb0B/IfQTDiKUF1O9aLJvwIzEUcVsbFSEgu+THr8NvBA9PfT7v7rxJ3u/kzF39Hon7eB1cBfzexid19Ry9e/NirDse5eHh33C2AqVSQxM7sAuACgBU1rRN6Lf+3McecuZ8Rdc7lv9HYUr8/h1F8uoUef8Iu+vOm0TmVFi1ZljHloFmWlcPtlvWv9fLPU392WXmtko1KLQRvL3X2f2h7f3T80s6+AfaNNBaSu9VTUwFYmxPWpJq4g4b5jNMLQa4gjeu1FCXGdkvZvNTUnBscTPvSK288S9v0tOdjM2pnZLWb2LaF9uITQYWnAgNq8sJnlRq/5dEUCA3D394DZVT3P3e93933cfZ98mlZX3eK5zbnlkj4MGLSe8e9+yeMff86u3y3i2T93BaBgaWwDn5q8/OblXDt+Nj37bOQ3p+3I8kXNan5SkjWFubTtuGXtuE37smh/E/ot7Wnetk5ibWk6m/unEu0GzHX3tQlx/cws+RfxbsBGNveBTQeaAzuliIPNA+Iq+r6SXzs5bqspiQWfufsHCbfE2s+iFPEPARcCdwM/JCShi6N9teswgC6EZsMlKfal2ibAWy914PTv7MZ5QwcyfMguXHLEzrRsVc7SBfksW1D7L1rZUm6eM/rPsxm4dxGjzuzH7C9b1vykFObMaEGP3htp3rJyFbnPzhvYWGwsnN00Pq+4JzunfA2zfYCdifrvCXPEtjOzgxJi2gFHR/tIiMsnjNSuiMsDfgpMikYmArxMSGqnJ730GYTv0VnR43eB5VXEFRBar2LRhH4C1Vmlf1Jm1oIwNHWMu9+VsH2LGfDABsLInsTnJ1ftlxNqct1TPL87MKcOZW4SysuNed+E3wydupdw0DGFPHVf13ouVeNg5lz1xznsfcBaRp/Vjy8/TD23Lx3vTmrPWSOWcOBRhbz6VPjnn5PrHHRMIR++0ZaSjU3kt7R7rBfFNLNHgVnAh0AhsDdhIvMC4A9R2HOEhDLBzEawebKzAbduLpp/bGZPAHdGc7hmARcRJkmfnhC31MzuAK42szXRa/+UMLDk2IS4EjMbTZjcvIAw2flQ4FzgUnePbTSPkljtNQdyCYkn0fAUsXOAPZK2HZX4wN3LzOx94CQzG5PQJ7Y/0JdGlMQOOLIQgAF7hgmu+x66msIVeaxakcenU8NI330OWU2LVuX02zVMWt5zyFradSplQ1EOH7zeDgg1hPNGLeSTqW0oWpPDDgM3cMolS5nzVQueGackFodLblzA0GNW8did3dhQlMMu31m3ad/yRfmbmhXT+bxmTm/J5H904MJrF5KX7yye24yjzlpBj94bueWSVN0wjVi8wxw+A04lrMTRClgMPAtc4+7LAdy93MyOAn4H3EtoKXoXOMTd5yUd7xzCBOXrgQ7ANOAId/8wKW4ksBb4JdCD0G9/srs/nxjk7n+yMDP+csIqIHOBS9z93q1/65vZliMlmw4zG05oGhzg7t/UYt+7hDbhKwg1qXOBwdG2Q9x9chR3DvAgcCfwArAXIdntCVzr7mOiuMOASVHMOMLoxGsJtbgZ7n5wde+jnXXy/e0HtXz32ffKwmkpt097pzW/PilMQ3n4vc/p0Tv59wEsnpfP2fuH5vScXGfMg7PYefB6WrcrY/mifCb/vQMT/9B9i2HcUjdVfQ4Aj9zenQm396g2LvHzAmjWopzhVy7ikOMLadOujJmfh2WnUi1Rta161Z/+b10GW1Ro22F7/86Bv0wr9o0Xfr1Vr9WUqCZWN6cC9wH3AOuBJwm/Sl5IinsY6E0YKPJz4E3CIJJKSdHdXzWz04ExhF9S3wC/io7ZaAzrtVeNMYlffFUpLzN+e/aOcRRJqpDO51CbuI0bcrj/2u24/9rttqZYDZsDMTYnStCkk5i7jyfM9artvtnAj1LssqS4csK6Y2Ori4tiHyfMR0u0xchIEWnAlMNi16STmIhINunKzvFTEhMRyZI4RydKoCQmIpIN9bOKfaOnJCYikgVhsrOyWNyUxEREskXresZOSUxEJEtUE4ufkpiISDaoTywjlMRERLIi3rUTJVASExHJFjUnxk5JTEQkGxxMAztipyQmIpItqonFTklMRCRblMNipyQmIpIlVq72xLgpiYmIZIOjyc4ZoCQmIpIFhmuycwYoiYmIZIuSWOyUxEREskVJLHZKYiIi2aA+sYxQEhMRyRKNToyfkpiISFa4mhMzQElMRCQbHCWxDFASExHJFrUmxk5JTEQkSzRPLH5KYiIi2aIkFjslMRGRbHCHMrUnxk1JTEQkW1QTi52SmIhItiiJxU5JTEQkGxwoVxKLm5KYiEhWOLj6xOKmJCYikg2OBnZkgJKYiEi2qE8sdkpiIiLZoiQWOyUxEZGs0ALAmaAkJiKSDQ7oUiyxUxITEckW1cRipyQmIpIVWnYqE5TERESywcE1Tyx2SmIiItmiFTtipyQmIpIt6hOLnZKYiEg2uGt0YgYoiYmIZItqYrFTEhMRyQrHy8rquxCNjpKYiEg26FIsGaEkJiKSLRpiHzslMRGRLHDAVROLnZKYiEg2uC6KmQlKYiIiWaKBHfEz15DPBs/MlgFz6rscGdIFWF7fhZC0NebPawd371rXJ5vZy4Tzk47l7n5EXV+rKVESk22amX3g7vvUdzkkPfq8JNty6rsAIiIidaUkJiIiDZaSmGzr7q/vAkit6POSrFKfmIiINFiqiYmISIOlJCYiIg2WkphgZsPNzM2s0Mw6Ju3Li/aNqYdyDTGz98xsXVSGwdkuQyr1dT7ilvC5p7odtg2Ub7KZTa7vcsi2TSt2SKL2wJXAVfVdkMhfgPXA0UAR8FX9FqfR+gkwP2nb5/VREJHaUhKTRJOAS83sTndfXJ8FMbMcYCBwg7u/VkNsc3cvzk7JGqWP3f2bmoJ0nmVbpOZESXR9dD+ypkAz28/MXjWztVFz37/NbL+kmPFmNt/M9jazN82syMy+NrMLazj2cKCM8O9zdNS8NTvpmEPM7B0zWw/cGu07xcxeM7NlUbk+MrOzk47dNzre8KTtB0fbD07Ylmtm15vZoqjsk81s95rOTWOQ0NQ41MyeMrNC4L1o375m9nT0Oaw3sxlmdqOZtUw6xmwzG5/i2Fs0x0af3ZdmVmxm083s+My9O2lMlMQk0SLgj8AFZrZDVUFmNgiYAnQEhgNnAe2AKWa2V1J4O+AxYAJwLPA+cJ+ZHVJNOV4EDoj+/gswBEj8UmsPTAQeB34UHR9gR+Bp4HTgOOB54IGakmY1xgC/AR6NjjcJeK6Ox9qW5UZ9nxW33IR9jwKzgJPY3MzcB/gYuBA4ArgLOBd4qC4vHvW/PQZ8DZwA3BYdc2BdjidNi5oTJdktwM+BawhfTKn8FigGfuDuhQBm9i9gdvS8ExJi2wK/cPfXo7g3gMOBU4HXUx3c3ZeZ2cro4Xx3n5oU0gY4w93/kfS8Gyv+jpojJwM9gYuAP1X1hlOJBrhcBtzv7ldEmyeZWRlwc22O1QB8mfT4beCB6O+n3f3XiTvd/ZmKv83MovjVwF/N7GJ3X1HL1782KsOx7uFaJWb2BTAVmFHLY0kTo5qYVOLuBcDtwFlmVtUv4aHACxUJLHreakIt5aCk2KKKBBbFFRN+cffZimKWAi8kbzSzAWb2uJktAEqi23nU7Rf9nkBr4Mmk7RPrcKxt3fHAvgm3nyXs+1tysJm1M7NbzOxbwo+ZEuARwIABtXnhqNa3LyFZbrrYlru/R/hRJFItJTFJ5Q6gABhbxf5OhKbHZIsJTYyJVqaIKwZa1Ll0sNTdK12YyczaAP8C9iI0ex1I+HJ8EGheh9foGd0vSdqe/Lgx+MzdP0i4JdZ+Un3ODxGaEu8Gfkg4zxdH+2r7uXYB8kl9XhvjuZaYqTlRtuDua83sJkKN7LYUIQVAjxTbe0T7Mi3VWmlDgB2AA939rYqNZpb8b3xDdN8saXvnpMcVX97dgekJ27vXrqgNXqVzbWYtCH2bY9z9roTte6Z47gaSzrOZdUqKWU6oyaU6r91pvNfJk5ioJiZVuRdYwOYRi4mmAEeaWduKDdHfR0f76kOr6L6kYkPUr3VsUtwSQk1wj6TtRyY9/gRYB5yctP2UrStmg9ccyCXhPEeGp4idw5bn+ajEB1GN+n3gpKgfEwAz2x/ou5VllSZANTFJyd2LzWwsqVclv47wZfRvM7uF8Gv9SkIiqaoJMtPeIQwuuMfMriH0Z40i/NJvXxHk7m5mTwA/M7OvCAMHjgQOTjyYuxea2R3ASDNbQxiZmNxf1OS4+yozmwpcbmaLCOf3XGC7FOETgQej8/gCoal3eIq4awjn9+9mNg7oShjsUa9zFaVhUE1MqvMQYRBGJe7+CeFLfzXwMKFTfy1wkLtPy2YBE8q0jDBAIZcwzP4mwgi7CSnCfwk8SxhC/wShH+fSFHFjgBuBMwmDVg4n1DabulOB/wL3AOMJyeaXKeIeZvNo1eeBYVSeKgGAu79KmBYxkPC5jAB+hUYmShp0KRYREWmwVBMTEZEGS0lMREQaLCUxERFpsJTERESkwVISExGRBktJTEREGiwlMdlmJFzDquK2xsymmdklKZaPivu1t7jOWHTtstm1PM7BZjYmcfWJmMo3xsxqnA9T1TW80j1+XOc54bPsG8fxRKqiJCbbop8Q1kI8EfgP8AfC5V+y7TpSTM6twcGECb76vyWSBVp2SrZFH7v7N9Hfk8ysP2EFh5SJzMzygVKPeea+u38b5/FEJH76tSgNwftAWzPrltDs9wszu9XMFhIW9O0AYGYnmNlUMysys0Ize8rMKl27zMxamdm9ZrbCzNaa2XPA9skvmqo50cxam9nNZvatmRWb2WIze8bMupvZGEItDKCkolk06XVvMbNZZrYxuh+Z3PRoZnub2ZtmtsHMFpjZaMK1umrNzLqa2Tgz+yo6J/PM7DEzS7XWIcCuZvZ6FLvIzMamKF8XM7svKluxmX1pZhfUpXwiW0s1MWkI+gFlhPUZK1arH0lIbhcQ1kvcYGYXAvcR1nwcS7iq9BhgipkNcvc10XPHAT8lLDL7PuGaWI/VVAgza0a4ZtlgwtqMUwmLCw8jXEftAUIy/BlwQFTmiufmAa8AuxGaKT8FvgeMJlyf7fIorgvwGmE9wrMJCXoEdb+IaCfCJVGuBpYBvaLXetvMdnH3DUnxfydcg+2m6H2NBsoJ5xEza0e4knPLaNusKO4+M2vu7n+oYzlF6sbdddNtm7gRVjh3wkKweYTE8HNCMvh7FNM3ivmQaO3PaHsbYBXwYNIx+wIbgV9FjwdGx7sqKe6+6LjDE7aNB2YnPD43ijmmmvcwJorJS9p+ZrR9aNL2kVH5ukWPb4ge90mIaU1YLd7TOIezgfHV7M8FekdlOT5FuZPPy5+BNUCH6PFoQlIckCJuecX7Tvgs+9b3vyvdGvdNzYmyLfqScL2qAsJ1zR4lJJBEf3f3xD6wIUA74FEzy6u4AfOj4w2N4vYnNKM/mXS8iWmU63Bgsbs/V5s3EzmCcH2td5LKN4lwZePvJbyPqe4+t+KJ7r6OsAp8nZjZRdEoz7VAKVBx7IEpwlOdlzZsvi7YEcB7wKyk9/EK4cKiu9W1nCJ1oeZE2RYdT0g+a4A5vmWTF2y+8nKFbtH9q1Ucc2V03zO6X5K0P/lxKp0JFwqti26EK08nX0wy8dgQyvdZiv3plG8LZnYpcDfwe0Kz5EpCEp9KuARNTa9T8biiD60b0J+a34dIViiJybboM988OrEqySMRV0T3w4HpKeIr+sMqkl93YGbC/u5plGs5W16pOF0rCP1HyVeKrjA7ul9URVnSKV8qpwD/dvfLKzaYWb9q4qs6LxXJewWwlNTXDwNdA0yyTElMGot3CImqv7s/XE3ce4SBCicDNydsPyWN15gEnGJmR7t7Vc17xdF9SzYnToCXCfPe1rr7l9W8xrvACDPr7e7zIIyIpO4X42xFuHhponOqiU91XtayuXb4MuEConPdfWkdyyQSGyUxaRTcfbWZjQDuMbOuwD8JAz22Aw4CJrv7Y+4+w8weAyqGjleMTvxxGi8zATgfeNzMbiIkxLaE0Xl3Rsnp8yj2cjP7J1Dm7h8Q+vXOAf5tZrcD04BmwE7AMcBx7l4E3AH8gjA/bgybRyeur+OpeRm40sx+Q5g4fihwUjXx5yecl2HAecAYdy+M9t9BGNn5ppndQah5tQZ2AQ5092PrWE6ROlESk0bD3ceZ2TzCl/5phAETC4A3gI8TQn9OqF1cQUgkr0Xxb9Vw/BIzO5wwF+yC6H4FYch5QRT2AmEwyi8Ik7ONMIqyxMyGAVdFz+0HrAO+BV4kjEjE3Zeb2Q+Au4CHo+P/ifB/tS6rlowlzKG7jNAHNoWQnGZWEX8sYYWU0YQfAdcTpgRUnINVZvb9qCxXEn4kFBKS2TN1KJ/IVrHKA7xEREQaDg2xFxGRBktJTEREGiwlMRERabCUxEREpMFSEhMRkQZLSUxERBosJTEREWmwlMRERKTB+n+gnd5I0JLOXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(pipe_lr, X_train_fold, y_train_fold, display_labels=['Non fraud', 'Fraud']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2XElEQVR4nO3dd5xcVf3/8dc7uymkJ5AEEggJBFAwgF9RiD8ERCUoXbpKlSIlIiJNWihKEQUVQhEFASkCiqBSjBQLCQpICxBaQggkpPe++/n9ce4mk9nZ3bthM7vZfT8fj/uYnXs/c++Z2WQ+e8o9RxGBmZlZS9KuuQtgZmZWzMnJzMxaHCcnMzNrcZyczMysxXFyMjOzFqeyuQtgH98GvSti0Cbtm7sY1ghvvty5uYtgjTSf2TMios+avn74F7vEzFlVuWKff3npYxGx55peqzVwcmoFBm3Snv88tklzF8MaYXj/7Zu7CNZIo+P+9z7O62fOquI/jw3MFVux0VsbfJxrtQZOTmZmZRBANdXNXYx1hpOTmVkZBMHyyNesZ05OZmZl45pTfk5OZmZlEARVni4uNycnM7MyqcbJKS8nJzOzMgigyskpNycnM7Mycc0pPycnM7MyCGC5+5xyc3IyMyuDINys1whOTmZm5RBQ5dyUm5OTmVkZpBkiLC8nJzOzshBVqLkLsc5wcjIzK4M0IMLJKS8nJzOzMkj3OTk55eXkZGZWJtWuOeXm5GRmVgauOTWOk5OZWRkEoop2zV2MdYaTk5lZmbhZLz8nJzOzMgjEsqho7mKsM5yczMzKIN2E62a9vJyczMzKxAMi8nNyMjMrgwhRFa455eXkZGZWJtWuOeXm5GRmVgZpQIS/cvPyJ2VmVgYeENE4Tk5mZmVS5fuccnNyMjMrA88Q0ThOTmZmZVLt0Xq5OTmZmZVBmvjVySkvJyczszIIxHJPX5Sbk5OZWRlE4JtwG8GflJlZWYjqnNsanV16VFJIuqxofy9Jt0iaIWmhpNGShpZ4fSdJP5E0RdJiSWMk7VIirp2kcyVNlLRE0kuSDqyjTMdLekPSUknjJX0n7/txcjIzK4Mg1ZzybI0l6XBguxL7BTwE7AmMAA4E2gNPStq4KPzXwPHAhcDewBTgMUnbF8VdCowErgO+CowF7pP0taJrHw/cBDyQXf8+YJSkk/K8JzfrmZmVydoYECGpJ3ANcDpwV9HhfYGdgd0j4sksfgwwATgL+G62bzvgG8CxEXFrtu9pYBxwSXYeJPUFfgBcERFXZ9d4UtIQ4Argr1lcJfAj4I6IOK8grj9wqaRbImJ5fe/LNSczszIIRHXk2xrpKmBcRNxd4ti+wIc1iQkgIuYCDwP7FcUtB+4tiFsB3AMMl9Qx2z0c6ADcWXSdO4GhkgZnz4cBfUrE3QGsT0qY9XJyMjMrgwCWR2WuLS9JOwNHAifXEbIN8GqJ/eOAgZK6FsRNiIhFJeI6AEMK4pYCb5eIA9i6II4S1y6Oq5Ob9czMykKNWc9pA0nPFTy/OSJuXu1sUntSn87VETG+jvP0BiaW2D8re+wFLMjiZtcT17vgcU5ERI44SpyzOK5OTk5mZmUQNGqGiBkRsUMDMWcD65H6duqi7NKl9q/tOOqIzcXJycysTJpqJVxJA4HzgOOAjgV9QmTPewLzSTWVUrWUXtljTc1mFjCwnrhZBY+9JKmo9lQqjuzaUwriehcdr5P7nMzMyiBCVEe7XFsOmwGdSAMOZhdskEbTzQaGkvp4tinx+q2BSRGxIHs+DhgsqXOJuGWs6mMaB3QENi8RB/BaQRwlrl0cVycnJzOzMkgDIipybTm8CHyxxAYpYX2RlFAeAgZI2rXmhZK6A/tkx2o8RLr/6eCCuErgUODxiFia7X6UlKy+WVSebwGvRsSE7PkYYEYdcbOAfzf0Bt2sZ2ZWFmqy6YsiYg7wVK0rSADvRcRT2fOHSIniTklnkmpU55L6hK4qON+Lku4Frs0GWkwATgIGU5BgImKapGuAcyXNB14gJbDdKRiaHhHLJV1Auun2A2B0FnMsMCIiljX0Hp2czMzKIA2IKO9igxFRLWlv4GpgFKkpcAzwxYh4vyj8GNLgisuAnsBLwJ4R8UJR3HmkEX6nARsC44FDIuLhomvfKCmAM4AzgUnAqRExKk/ZnZzMzMpkbS+ZEVE7+0XELFKN5dgGXrsY+H621RdXRUpgl9UXl8XeRBru3mhOTmZmZVAzQ4Tl4+RkZlYm1R6DlpuTkzWpH35jM55/qjuHnzaVo8+eunL/xPGduP2qDXn9hS4snNeOfpssY/hhszjguOlUZP8KP5rcnlHnb8y749ZjzsxKOq1XzaBPLOGQUz7is7vPX3muye905OHbNuClZ7oy5b0OdO5azZbbLeLIs6aw+TZLVivP1d/bhDde6MLMqe2proaNBi3jq4fPZO+jZ1Dhdd8abYONlnHIKdPYYtvFbLbNYjqtFxz5uU/y0eQOq8V17bGC4y6Ywuf3nEvHTsHrz3fmxov6M/GN9Zqp5M0vApZXOznl5eRkTebJP/Zkwmu1v3xmTq3krAOHsP6Gy/nOxR/Qo/cK/vevrtxyaX/mzKjkuPPTPXqLF1bQo/cKjjp7ChtstJxF89vxyF3rc/63NueCWyaw89fmAvD809146ZmufPngWQwZupiF8yq4b1RfTtt7S67501tsse3ilddetqQd+x07nY0GLUPAc09344YLB/DhxI6cdOkHZflcWpP+g5axyz5zeevl9Xj12S7ssNuCElHBxbdNpN8myxh1/gAWzKng0BHTuOr+dzj5K1syY0qHEq9p/VKznpNTXmVNTpKOBm4F5gKDI2J2wbFK0qy4F0fEyDKXaxhwLfApoDPw6Yh4sZxlKCUb6VL2z2NNLJhbwU0jB3DiyA+44pRBqx17dnQP5s6q5Gd/eouNN0+3S2y/8wKmTOzI3+/vvTI5DdpqCd//2eoDiHb88jyO3HFrHr+n98rktNt+s9n3mBmooPl++/83nyN33Jo/3tKHs34xaeX+H9743mrn+8xu85k1tT2P3dPbyWkNvDK2C4dtl+6r3PMbM0smp2HD5/GpHRdy1kGb89IzaV7R157vwu1jX+fgk6dzwwUDylrmlqSpZohoC5orjfcgzQvVUvyalKj3IU31/mbzFmfdc8tlG7HpVkv44gFzah1bviz9h+zcrWq1/V16VFFdXf95KyqhS/cqKtuvmimlx/pVqyUmgC7dqxmw2VJmTm3fYFm79VpBReUaT/nVppUYDFbLTnvMY8aUypWJCWDR/ArG/q07w4bPXZvFa9FqhpKvhSUzWqXmSk6PAyMkbdhM119JUjtgK+AvEfFERIwtMW18TWzHUvvbulef7cLo+3tz6o8nlzy+yz5z6NF7BdeftzFTJ3Vg4fx2/PuRHvz9/l4ceOL0WvHV1VC1AmZNq+R31/Tjg3c7ss/RM+otw7zZFUx8oxObDFlS61hEOt+CuRX88y89GH1fb75+Qu3rWtPYdMslTBzfqdb+997sRL+Nl9Opc1WJV7UFTTp9UavXXJ9Czfj48+qNAiR9LlvzfoGkhZL+LulzRTG3SZos6dOS/ilpkaS3GlqvPmtmrCJ9DhdICkkTi845TNIzkhaT3VEt6TBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUml5sNqcVYsh5+fvTEHfWcamwxZWjKmV58VXPPwm0x6qyNH7bQ1X99qWy49bhCHnDKNQ06ZViv+lkv787WB23P49p/ivlF9OfeG9/j0F0r1bawy6vwBEOKA42snnWdHd+drA7fnwE8O5UcnDGLfY2fwzdM/WrM3bA3q1nMFC+bW7jGYP6ciO95WkxNUo1ybNd+AiCmk9ee/J+nqiHivVJCkbYGnSZMEHk2qGZ8DPC1pp4h4qSC8O2mJ4mtJywofA9wgaXzhKpBF/kJakfFfpKa9W0gLadXoQVoJ8mrgh0BNT/tmwP2kZYmrgV2AWyStFxE35vwMCo3Mzv8zUq1yB1af96rF+v31/Vi2pB2Hf7fuL/s5Myu49LjBdOpczfm/mkD3XlW8+K+u3P3zfrTvEBx66uoJ6oDjp7Pb/rOZNa09o+/vxRWnbMr5N09kp6/MK3n+e37Zlyf/2Jvv/3QSAwbXnhVl6I4L+OUj41k4r4L//asbD9zYByk45pypJc5mH5eUaqu19pe/KC1KGq3nIaJ5NedovSuBE4GLqPvO5QtJyeJL2VxSSPobafGsi4CvF8R2A06uSUSS/gHsARwOlExOETFdUs2gjMkRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEWkuqkYlJ0m9gNNJi4n9INv9uKQqUvKr63UnACcADBzQPL/GaZPbc/cv+nH61e+zfFk7lhfkheVL27FgbgXrda3ivuv78dH7Hbj9P6+t/Kt5u88voLoabv/Jhux5+Ex6rL/qr+k+/ZfTp/9yYDE7fWUeZx44hF9d0r9kcvrz7etz6+X9OersKQw/vPQs/F26V7Pldunvik9/YQHt2wd3XduPfY6ayQYbLW+6D8QAmD+nkm49V9Ta3zX73dfUoNoa34TbOM3WuJlNqfFT4EhJW9URtgvw55rElL1uHqlWsWtR7KLCGlI2i+5blF6jJK8VwJ+Ld0raQtLd2YSGy7PtOFLfVWMNBboAvy/af099L4qImyNih4jYoc/6zfOffcqkjixb0o4rT92UAz85dOUGcP+NfTnwk0OZ+Pp6THijE/0HLa3VnLPV9otYsbwdH06svytvi20XlYwZfX8vrjt3Yw48cRrfOC1/M90W2y2iulpMndQ2hzSvbe+92ZFNt6zdxDtwiyV8NLk9Sxa1zeQEbtZrjOa+z+kaYASpGa54anWovVBVjamsWtyqRqklhpeSJjpcU9OyeaRWktQV+BuwiNTE+A5pCvmTaGDuqjpslD0Wf7u2+E6RzbdZzFX3v11r/1kHDeFLB85i+OGz6D94Kb37ruD157owf07Fagnqjf91AWD9DeuuvVRXw7j/dqH/pqt/2f37kR789PSB7PmNmZxw0YeNKvcrY7oiBRttWrqPzD6eMY/3YPhhsxm60wJeGZtG7HXuWsVOX5nHkw/2bN7CNaPmmPh1XdasySkiFki6nFSD+kmJkFmkWW+LbUiOlRSbQKnxxsOATYEvRMS/anZm92kVqhk2Vvzn+fpFz2uSbz9WLdBV87xF69qjiu0+X3qgQt+Nl608ttcRM3jiD7344eGbc9BJ0+jeawUvj+nKAzf24f99dQ59B6TkdMfVGzJ/TgVbf3YhvfsuZ9a09jx2d2/G/68z51y/qlvylbFduPzkTdnsk4v5yiGzeP35Veujte8QDBmamvCeHd2dx+/tzU5fmUufActZvKAd/32yO4/cuT5fO2Im629Yu+nJGrbzXnMA2CL7nD+7+zzmzKxk7sxKXhnblbGPdee15zpz9nWT+NWlG628CVeC+0b1bcaSNz+PxMuvuWtOkKZx/z6lZ7h9GthLUreImA8gqRvpfqSnylbC1dV8E678cz/rN9qvKO4jUs3tU0X79yp6/jKwEDgEeKJg/2Efu6QtxCc/s4if/vEtfnfNhtx44QAWzk/TF33z9I848MRVgyGGDF3EH2/pw1N/6smi+RX06rOCzbZezE//+DbbfG7hyrgX/9WN5Uvb8farnfn+fluudq1+Gy/j9v+kRTY32nQpUQ23XbkRc2dW0qV7FQMGL+XMX0xit/1LVbQtjwt+tfr4pRFXpJuZX3qmC2cdNIQIceGRgzn+wg8Z8eMPaN+pmtef68JZB2/O9A/bblNqhFjh5JRbsyeniFgq6RLg5hKHLwX2Bv4u6UpSTeZsUoK4pHylXM0zwDzgekkXkfqLziet+tijJigiIlu869uS3iStebIXsFvhySJiTrZ413nZ4l2PA58Fvl2G97JWPPbhi7X2ffIzi7jsznfrfd2w4fMYNrz0iLxCR/xgKkf8oOGRdgO3WMqFv57YYJw1zvD+2zUYM39OJT/7/kB+VobyrEvcrJdfS0njt5IGL6wmIl4mfZnPA34L3EFa5GrXomHkZRMR04EDgArScPLLSUPQ7ywRfhrwB9JQ8XtJ/V8jSsSNBH4MHEEa7LEHqXZoZq2EZ4honLLWnCLiNuC2EvtXAFsW78+OPQt8uYHzHl3H/t1ylGkFJW7BqOuc2bEngE+XODSyKG4OKeEUU1FcFan2dX59cWa2bnPiya/Zm/XMzNoC3+fUOE5OZmZl4nuY8nNyMjMrgwhY4cUGc3NyMjMrEzfr5efkZGZWBu5zahwnJzOzMsmzWKMlTk5mZmXiARH5OTmZmZVBhPucGsPJycysLESVR+vl5uRkZlYm7nPKr87kJKma0ktGlBIR4URnZlYHr+fUOPUllEvIn5zMzKw+kfqdLJ86k1NEjCxjOczMWj2P1suv0U1x2TLl6wMfRkTd62ubmdlK4QERjZL7k5K0t6QXgLnAu8DQbP8tkr6xlspnZtZqROTbLGdykrQ/8CfSaq9ns/o6QxOAo5q8ZGZmrUyEcm0NkTRc0hOSpkpaKmmypN9L2roorldWgZghaaGk0ZKGljhfJ0k/kTRF0mJJYyTtUiKunaRzJU2UtETSS5IOrKOMx0t6IyvfeEnfacRHlbvmdBFwa0TsAVxbdOxV4FONuaiZWVuTakVNk5yA3sDzwKmklbPPBbYBxkraFECSSCtr70lagftAoD3wpKSNi873a+B44EJgb2AK8Jik7YviLiUtqnod8FVgLHCfpK8VBkk6HrgJeCC7/n3AKEkn5XlzkL/P6ZPAWdnPxZXO2aQ+KDMzq0dTDSWPiLuBuwv3SfoP8AZwEPBTYF9gZ2D3iHgyixlDau06C/hutm874BvAsRFxa7bvaWAcadT2vtm+vsAPgCsi4urssk9KGgJcAfw1i6sEfgTcERHnFcT1By6VdEue8Qp5a07zgA3qODYImJ7zPGZmbdZa7nOamT3WfPHvSxq49uSq68dc4GFgv4LX7Zu95t6CuBXAPcBwSR2z3cOBDsCdRde9ExgqaXD2fBjQp0TcHaSKzM553kze5PQ34FxJPQv2RVboU4FHcp7HzKxNCkR1dbtcW16SKiR1kLQFqRltKimpQGrme7XEy8YBA7OR1zVxEyJiUYm4DsCQgrilwNsl4gC2LoijxLWL4+qVt1nvPOA/wHhS1S2Ac4BtgR7A/jnPY2bWZjWiUrSBpOcKnt8cETeXiHsW+Ez289ukJrxp2fPewMQSr5mVPfYCFmRxs+uJ613wOCeiVt2uVBwlzlkcV69cySkiJkr6P+BiUtWuCtgFeBS4MCI+zHMeM7M2Kxo1t96MiNghR9wRQHdgM1J/0N8k7RwRE0mjqkvlw+JCrI046ojNLfdNuBExGfj2x7mYmVmb1sT3MEXE69mPz0p6hFRTOgf4DqmmUqqW0it7rKnZzAIG1hM3q+CxlyQV1Z5KxZFde0pBXO+i4/Vq9O3KkvpL+mw28sLMzHJqwqHkJc4dc0hNezV9RONY1f9TaGtgUkQsKIgbLKlzibhlrOpjGgd0BDYvEQfwWkEcJa5dHFevxswQcaSkCcD7pLHt70uaIOlbec9hZtZWBVBdrVzbmpDUD/gE8E626yFggKRdC2K6A/tkxyiIaw8cXBBXCRwKPB4RS7Pdj5KS1TeLLv0t4NWImJA9H0OasKFU3Czg33neT65mPUmnAr8ARpNuwvoI6AccDvxWUo+IuD7PuczM2qQAmug+J0l/BF4AXibd6rMlcDqwgnSPE6SkMwa4U9KZpGa8c0l9QletLFbEi5LuBa6V1J50H9RJwGAKEkxETJN0DWnk9vzs+ocCu1MwND0ilku6gHTT7QekvLE7cCwwIiKW5XmPefuczgBui4hji/b/RtJtpI44Jyczs3o04bx5Y4FDSN/NHUgtWk8Bl2eDIYiIakl7A1cDo4BOpGT1xYh4v+h8x5BunL0M6Am8BOwZES8UxZ1HGuF3GrAhaQT3IRHxcGFQRNwoKbLynQlMAk6NiFF532De5LQhq8bOF7uL9CGZmVl9mig5RcSVwJU54maRaizFFYviuMXA97OtvrgqUgK7LMe1byLde7VG8ianV6jdCVZjC0rf6GVmZiut+WCHtihvcjoNuEfSDOAPEVElqYI0keCZwGFrq4BmZq2Gl8PIrc7kJOl9Vv8oe5Ca9qokzSaNba8gtT/eC2y6FstpZrZuC4g1HInXFtVXc/o7zvNmZk3IySmvOpNTRBxdxnKYmbV+/nM/t9zTF5mZ2cfk5JRbo5JTtijVVqTx8quJiNubqlBmZq1OE96E2xbknSGiJ/AXYKeaXdlj4d8BTk5mZvVowptwW728c+v9mLSC4S6kxHQAaTqK3wHvAp9bK6UzM2tNqpVvs9zJaTgpQY3Nnk+OiKci4kjSvEmnrY3CmZm1Jop8m+VPThsB72ZTVywBuhUc+wOwV1MXzMysVYlGbJY7OU0lTQYI8B4wrODYkFrRZmZWRGlARJ7Nco/W+xcpIf0ZuAO4SNIg0vTsR7H62iBmZlaKa0W55U1OFwM1K9/+hDQ44lCgMykxjWj6opmZtTLVzV2AdUeu5BQR75CtrhgRy0lrdJyxFstlZta6+D6nRvEMEWZmZeKRePnVNyv5hY04T0TEpU1QHjOz1svJKbf6ak4jG3GeAJyczMysSdQ3K3neYebWzN58uTPD+2/f3MUwswa4WS8/9zmZmZVD4KmJGsHJycysXFxzys3JycysTNysl5+Tk5lZuTg55ebkZGZWLk5OuTk5mZmVgZfDaJzGLtO+LWnBwfWBmyJiqqQhwEcRMX9tFNDMrNXwaL3c8i7T3hG4E/g6aSXcAB4mLaVxFfAmcM5aKqOZWavgmlN+eW+0/RHwZeAIoB8pQdV4hLRSrpmZ1ceLDeaWt1nvcOD8iLhLUkXRsQnAoCYtlZlZa+M+p0bJm5zWB16v41g7oGPTFMfMrBVzcsotb7PeBFZfmr3Q54DxTVMcM7PWS9X5NsufnG4HzpH0TaBDti8kfRE4HfjN2iicmZm1TXmT01XAX4A7gFnZvn8Bo4FHI+KXa6FsZmatSxMNiJB0kKQHJL0nabGk8ZIul9StKK6XpFskzZC0UNJoSUNLnK+TpJ9ImpKdb4ykXUrEtZN0rqSJkpZIeknSgXWU8XhJb0hampXvOw2/s1VyJaeIqIqIw4BdgZ8CtwC/AHaPiG825oJmZm1SrLoRt6Ethx8AVcAPgT2BG4CTgL9JagcgScBD2fERwIFAe+BJSRsXne/XwPHAhcDewBTgMUnbF8VdSlrr7zrgq8BY4D5JXysMknQ8cBPwQHb9+4BRkk7K9e5o5E24EfFP4J+NeY2ZmWWabkDEPhExveD505JmAb8FdgOeAPYFdiZVIp4EkDSGNIbgLOC72b7tgG8Ax0bErdm+p4FxwCXZeZDUl5QUr4iIq7PrPplNxHAF8NcsrpJ0+9EdEXFeQVx/4FJJt0TE8obeoBcUNDMrlyZq1itKTDX+mz0OyB73BT6sSUzZ6+aSJlDYr+B1+wLLgXsL4lYA9wDDs0kYIN3P2oE0IUOhO4GhkgZnz4cBfUrE3UEa+b1zQ+8PciYnSdWSqurb8pzHzKytEmt9tN6u2WPNbT/bAK+WiBsHDJTUtSBuQkQsKhHXARhSELcUeLtEHMDWBXGUuHZxXL3yNutdQu18vj6wB+kep9tynsfMrG1q3E24G0h6ruD5zRFxc13BkgaQvqdHR0TN63oDE0uE1wxq6wUsyOJm1xPXu+BxTkQUv4tScZQ4Z3FcvXIlp4gYWWp/NlvEw8DcPOcxM2vT8ienGRGxQ57ArAb0J2AFcEzhoTquWDz77NqIo47Y3D5Wn1NEVAGjgO99nPOYmbUJTTy3nqROpBF5mwHDI2JyweFZlK6l9MoeZ+eMm1Xw2CsbBdhQHCXO2bvoeL2aYkBExxKFMDOzIk04lBxJ7UlDtT8HfC0iXikKGceq/p9CWwOTImJBQdxgSZ1LxC1jVR/TONL3/eYl4gBeK4ijxLWL4+qVd0DEwBLbEEn7k4YQPtfAKczMrOluwm0H/A74ErBfRIwtEfYQMEDSrgWv6w7skx0rjGsPHFwQVwkcCjweEUuz3Y+SklXxva3fAl6NiAnZ8zHAjDriZgH/bvgd5h8QMZG62xrfAU7JeR4zs7YpmnTevOtJyeRHwEJJOxUcm5w17z1EShR3SjqT1Ix3Lul7+6qVxYp4UdK9wLVZbWwC6YbewRQkmIiYJuka4FxJ84EXSAlsdwqGpkfEckkXkG66/YA0k9DuwLHAiIhYlucN5k1Ox5TYtwR4D/hv1vdkZmb1abqbcL+aPZ6XbYUuBkZGRLWkvYGrSWMDOpGS1Rcj4v2i1xxDSnSXAT2Bl4A9I+KForjzSCP8TgM2JE36fUhEPFwYFBE3SgrgDOBMYBJwakSMyvsGG0xO2Yi8F0k3c5W68cvMzHJoqvWcImJQzrhZpBrLsQ3ELQa+n231xVWREthlOa59E2kKozWSp88pSH1Kn17Ti5iZGV4JtxEarDllVcP3gS5lKI+ZWevkxNMoeYeS3wR8T1KHBiPNzKwW0bRDyVu7vAMiupHGtr8r6VHSdOqFH2FExEVNXTgzs9bEiSe/OpOTpHeBAyLiJdKaITVKdawF4ORkZlYfJ6fc6qs5DSLdDUxEeGkNM7OPy8kpt0YtNmhmZmvI/UmN0lBy8kdpZtZU/I2aW0PJ6WJJM3KcJyLiqKYokJlZa9WE0xe1eg0lp+1JKx82xH8PmJk1wM16+TWUnPaPiP+UpSRmZq2Zb8JtFA+IMDMrFyen3JyczMzKoGaGCMvHycnMrExU7eyUV53JyTfempk1Ifc5NYprTmZmZeJmvfycnMzMysXJKTcnJzOzMnHNKT8nJzOzcnFyys3JycysHMLTFzWGk5OZWRn4PqfGcXKyZvOZXedxyCnTGbjlErr2qGLuzEpef64zd/x0Qya91WllXNceKzjugil8fs+5dOwUvP58Z268qD8T31ivGUvf9mw7bAE/eeCdWvsXzG3HgZ8cWvI1373yffY6YhZ/f6AnV43YdG0XseULZ6e82nRyknQ0cGsdh78SEaPLWJxaJD0FEBG7NWc51pZuvap465X1ePi36zN3ZiV9Byzj0FOnce2f3+I7u2/FtA86AMHFt02k3ybLGHX+ABbMqeDQEdO46v53OPkrWzJjSofmfhttzvXn9+fNFzuvfF61QiXjtt5hIbt/fQ4L5/mWyRquOeXXppNTgYOByUX7XmuOgrQlTz3Yi6ce7LXavvEvdubX/xzPF/aewwM39WXY8Hl8aseFnHXQ5rz0TFcAXnu+C7ePfZ2DT57ODRcMaI6it2nvv9WJN17oUm9MRWVw2lWTufvnffnaETPLVLIWzjfhNor/pElejIixRdu84iBJHZujcG3JvNnp76UV2V/jO+0xjxlTKlcmJoBF8ysY+7fuDBs+t1nKaA07+KRptKsIHripT3MXpUVRdb7NnJzqJOloSSFpF0n3SZoDPJsd+6yk+yVNlrRY0nhJP5a0XtE5Jkq6rcS5Q9LIon2HSXpD0lJJ4yQdsPbeXcvSrl1Q2b6a/oOXctqVk5n5USVPPdgTgE23XMLE8Z1qvea9NzvRb+PldOpcVebS2tnXTeKv77/Efa++yjnXv0efActWO77Rpks5/Hsfcd25G7Niub9iCjk55edmvaRCUuFnUVj5/h1wN3AQqz6vgcCLwG3AfGAb4EJgM+Cwxl5c0peBu4C/AGcAfYCfA+2B8Y0937rm5395iy23WwzAB+924OyDN2fuzPYAdOu5go8md671mvlzKrLjVSxZVFG+wrZhC+e34/4b+/DymC4sml/BkE8t5rDvTuPah97i5D22XPk7++6Vk/n3X3usVts1smY9t+vl5eSUvFH0/N/ALdnP90fEWYUHI+KBmp8lKYufB9wu6ZSIaGwj+8VZGfaLiOrsvK8DY6kjOUk6ATgBoBO1v7zXJVeNGEjnblVstOkyDvrOdC6/513O2H8IH03ugFT6/3PpLnhbm955tTPvvLrq39orY7vyyrNd+MVf3mL/b8/gt1dtxO5fn82W2y3iuF0+0Ywlbbk8ICI/17mTA4DPFmzfLjj2x+JgSd0lXSnpHdIy9suBO0jfmVs05sKSKrJr3l+TmAAi4llgYl2vi4ibI2KHiNihPet2V9j7b3di/P+68NSDvTj7kM1Zr0sVh546DYD5cyrp1nNFrdd07VmVHXetqTm9/UpnJr/bkS23X0SnzlWcOPJDfn99X5YtbUeX7lV06V5Fu3ZQ2T7o0r2Kiso2/u0cOTdzzSnzakS8XbhD0rDsxykl4m8FvkxqynsRWAh8DrgeqN1BUr8NSM13H5U4Vmpfq7ZwXgUfTuxI/8FLAXjvzY783y4LasUN3GIJH01u7ya9FkACQvToXUXPDVZw7A+ncuwPp64W03fAXHbddy4jjx3EmEd7NE9Bm5lvwm0cJ6eGrfbPSVInYD9gZET8vGB/qbsQlwCr3YgjqXdRzAxSzatfidf3A95bgzKvs3pusJxNhizliT/0BGDM4z0Yfthshu60gFfGpj6Mzl2r2Okr83gyGzRhzWeLbRcxYLOl/OPhnsyaXsmZB25eK+bcG95j4huduPvn/UoObmkzIrzYYCM4OTVeR6CClFAKHV0i9j3gU0X79i58EhFVkv4LHCRpZEGf047AIFpxcrrw1xN4+5XOTHi9E4vmVzBgs6V8/YTpVK2AB27qC8DYx7rz2nOdOfu6Sfzq0o1W3oQrwX2j+jbzO2hbzr7uPaZO6sDbr6zHgnlpQMShI6Yxc2p7/vSbDVi+tB0vj6k9CGLZUjF7emXJY22Oc1NuTk6NFBFzJY0FzpA0hVTzORYodTfoPcBvJF0D/BnYjtJJ7CLgceBBSTeRRutdDEwtEdtqvPFCF3bZZw4HnjiNyg7B9A878PIzXbj3l/34aHKqcEaIC48czPEXfsiIH39A+07VvP5cF846eHOmf+jZIcpp4hud2G3/Oex37Aw6rlfN7Ont+fdfe3DH1Rsyb5a/SvJoymY9SRsDZwM7kL5b1gMGR8TEorhewE+A/bOYMcDpEfFKUVwn4FLgW0BPUpfF2RHxj6K4dtl1TwQ2JA3auqRwoFhB7PGkEciDSX3o10TEjXnen/9FrZnDgRtIfUyLgd8Dp5ESUKHfApuQBlicCPyTNPhitf6tiBgt6ZvASOAP2fHvZedstX5/fV9+f33DtZ/5cyr52fcH8rMylMnqdu91/bj3ulKtz/U7aset10Jp1kEBNG2z3hDgEOB50nfLHsUB2Wjih0jJYQQwGzgXeFLS9hFRODPOr4G9gDOBd4FTgMckDYuIFwviLgV+AJyXXfsw4D5Je0fEXwuufTxwE3A5MBr4EjBKkiLihobenMLj7td53dU7dtSXmrsYZq3a6Lj/+YjYYU1f363HxvF//++7uWL/8cjZDV5LUruCboDjgF9RVHOStB/wILB7RDyZ7esBTADujIjvZvu2I9WUjo2IW7N9lcA4YHxE7Jvt6wu8D1wRERcVXOfvQJ+I2LbgtR8Cj0TEUQVxvwH2BTaKiOKukdV4KLmZWZko8m15FN56Uo99gQ9rElP2urnAw6SBXYVxy4F7C+JWkLomhhdM3TacNMjrzqLr3AkMlTQ4ez6M1D1RHHcHsD6wc0MFd3IyMysTVUeurQltA7xaYv84YKCkrgVxEyJiUYm4DqQmxJq4pRR1TWRxAFsXxFHi2sVxdXJyMjMrh7w34KbctIGk5wq2E9bwqr1J/UzFZmWPvXLG9S54nBO1+4NKxVHinMVxdfKACDOzMkg34eauFc34OP1bRZctddHiGcDWRhx1xObimpOZWblU59yazixK11Jqakyzc8bNKnjslY0CbCiOEufsXXS8Tk5OZmZloohcWxMax6r+n0JbA5MiYkFB3GBJxbNIbw0sY1Uf0zjSRATFU4HU9CG9VhBHiWsXx9XJycnMrBwa1+fUVB4CBkjatWaHpO7APtmxwrj2pFXBa+IqgUOBxyNiabb7UVKy+mbRdb5FmqN0QvZ8DGmCglJxs0grOdTLfU5mZmXR9HPrSToo+/Ez2eNXJU0HpkfE06SkMwa4U9KZrLoJV8BVK0sW8aKke4FrJbUn3Qd1Eunm3W8WxE3LZrw5V9J84AVSAtudgqHpEbFc0gWkm24/IN2EuztpNp0REbH6CpUlODmZmZVL0096cF/R81HZ49PAbhFRLWlv4OrsWCdSsvpiRLxf9NpjgB8Bl5GmL3oJ2DMiXiiKOw9YQJrBpmb6okMi4uHCoIi4UVKQpi86E5gEnBoRo8jBycnMrByi6Zdgj4gG192MiFmkGsuxDcQtBr6fbfXFVZES2GU5rn0TaQqjRnNyMjMrF08Xl5uTk5lZuTg35ebkZGZWJqpu4na9VszJycysHIKmvsG2VXNyMjMrA9HkN9i2ak5OZmbl4uSUm5OTmVm5ODnl5uRkZlYO7nNqFCcnM7My8Wi9/JyczMzKItys1whOTmZm5RA4OTWCk5OZWbm4VS83JyczszLxfU75OTmZmZWLk1NuTk5mZuUQAVVu18vLycnMrFxcc8rNycnMrFycnHJzcjIzK4cAqp2c8nJyMjMri4Bwn1NeTk5mZuUQeEBEIzg5mZmVi/uccnNyMjMrFyen3JyczMzKwhO/NoaTk5lZOQTgJTNyc3IyMysX15xyc3IyMysLT1/UGE5OZmblEBC+zyk3Jyczs3LxDBG5OTmZmZWL+5xyc3IyMyuHCI/WawQnJzOzcnHNKTcnJzOzsgiiqqq5C7HOcHIyMysHL5nRKE5OZmbl4qHkuTk5mZmVQQDhmlNuTk5mZuUQXmywMZyczMzKxAMi8lN4aOM6T9J04L3mLsdasgEwo7kLYbm15t/XphHRZ01fLOlR0ueTx4yI2HNNr9UaODlZiybpuYjYobnLYfn492VNpV1zF8DMzKyYk5OZmbU4Tk7W0t3c3AWwRvHvy5qE+5zMzKzFcc3JzMxaHCcnMzNrcZycDElHSwpJcyT1KjpWmR0b2QzlGibpWUkLszJsX+4ylNJcn0dTK/i9l9q+3ALK95Skp5q7HNY8PEOEFeoBnA2c09wFyfwaWAzsAywC3mze4rRaBwOTi/a91hwFMavh5GSFHgdGSLo2IqY2Z0EktQO2An4UEU80ENsxIpaWp2St0osR8XZDQf6crZzcrGeFLssez2soUNLnJI2WtCBrdvu7pM8VxdwmabKkT0v6p6RFkt6S9J0Gzn00UEX693lB1sw0seicwyQ9I2kxcFV27DBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUnbNPTZtAYFTX67SLpP0hzg2ezYZyXdn/0eFksaL+nHktYrOsdESbeVOHetZtHsd/eGpKWSxkk6YO29O1sXODlZoSnAdcAJkjatK0jStsDTQC/gaOBIoDvwtKTtisK7A3cBdwL7Af8FbpD0xXrK8Rdg5+znXwPDgMIvqx7APcDdwFez8wNsBtwPfBPYH3gYuKWhZFiPkcAPgd9l53sceGgNz9WSVWR9izVbRcGx3wETgINY1dw7EHgR+A6wJ/Bz4Fjg1jW5eNa/dRfwFvB14CfZObdak/NZ6+BmPSt2JXAicBHpC6eUC4GlwJciYg6ApL8BE7PXfb0gthtwckQ8mcX9A9gDOBx4stTJI2K6pNnZ08kRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEXAScGNdb7iUbGDI6cDNEfGDbPfjkqqAKxpzrnXAG0XP/w3ckv18f0ScVXgwIh6o+VmSsvh5wO2STomImY28/sVZGfaLSGtKSHodGAuMb+S5rJVwzclWExGzgJ8CR0qq6y/XXYA/1ySm7HXzSLWKXYtiF9UkpixuKekv5IEfo5grgD8X75S0haS7JX0ALM+241izv8CHAl2A3xftv2cNztXSHQB8tmD7dsGxPxYHS+ou6UpJ75D+SFkO3AEI2KIxF85qaZ8lJcGVix1FxLOkP3asjXJyslKuAWYBl9RxvDepCbDYVFJTX6HZJeKWAp3WuHQwLSJWWxhHUlfgb8B2pOanL5C+9H4DdFyDa2yUPX5UtL/4eWvwakQ8V7AV1lZK/Z5vJTXp/QL4CulzPiU71tjf6wZAe0p/rq3xs7ac3KxntUTEAkmXk2pQPykRMgvYsMT+DbNja1upObeGAZsCX4iIf9XslFT8b3xJ9tihaP/6Rc9rvpT7AeMK9vdrXFHXeat91pI6kfoOR0bEzwv2Dy3x2iUUfc6SehfFzCDVvEp9rv1oveuUWQNcc7K6jAI+YNUIvkJPA3tJ6lazI/t5n+xYc+icPS6v2ZH1G+1XFPcRqeb2qaL9exU9fxlYCBxStP+wj1fMdV5HoIKCzzlzdInY96j9Oe9d+CSrAf8XOCjrJwRA0o7AoI9ZVluHueZkJUXEUkmXUHqW6UtJXzJ/l3Ql6a/rs0kJoq6mwLXtGVKn/PWSLiL1F51P+su8R01QRISke4FvS3qT1OG+F7Bb4ckiYo6ka4DzJM0njdQr7o9pcyJirqSxwBmSppA+32OBASXC7wF+k32OfyY1uR5dIu4i0uf7oKSbgD6kQRLNeq+dNS/XnKw+t5IGL6wmIl4mfZnPA35L6gxfAOwaES+Vs4AFZZpO6tivIA0nv5w04uzOEuGnAX8gDRW/l9RPMqJE3Ejgx8ARpMEee5Bqh23d4cDzwPXAbaQkclqJuN+yavTmw8BwVr8lAICIGE0a/r8V6fdyJvA9PFKvTfOSGWZm1uK45mRmZi2Ok5OZmbU4Tk5mZtbiODmZmVmL4+RkZmYtjpOTmZm1OE5O1mKUWDZ8vqSXJJ1aYhqipr52rXWesrWjJjbyPLtJGlk420ETlW+kpAbv+6hrDaW852+qz7ngdzmoKc5nbY+Tk7VEB5PmyjsQ+A/wS9IyHeV2KSVuGm3AbqQbT/1/y+xj8PRF1hIVLhv+uKQhpBkDSiYoSe2BFdHEd5RHxDtNeT4zy89/3dm64L9AN0l9C5rfTpZ0laQPSRO59gSQ9HVJY7Nl1edkS4yvtnaUpM6SRkmaqbSc+0PAxsUXLdWsJ6mLpCskvZMtKT5V0gOS+mVLj1+UhS6vaZ4suu6VkiZIWpY9nlfcBKhVy9ovkfSBpAtIayU1mqQ+km6S9Gb2mbwv6S5JpebCA/ikpCez2CmSLilRvg0k3ZCVbanS8uonrEn5zOrimpOtCwYDVaT5+2pmHz+PlLROIM2nt0RpOfYbSHMCXkJahXckafn4bSNifvbam4BDSZOL/pe0JlHNUu91ktSBtGbU9qS5+8aSJpUdTlrH6hZSkvs2aZn5qoLXVgKPAVuTmgtfAXYCLiCtj3VGFrcB8ARpvrqjSIn3TNZ8ccbepKUrzgWmA/2za/1b0iciYklR/IOkNbAuz97XBUA16XNEUnfSyrfrZfsmZHE3SOoYEb9cw3KarS4ivHlrERtpxuogTQBaSfrCP5H0Jf9gFjMoi3mBbG7IbH9XYC7wm6JzDgKWAd/Lnm+Vne+corgbsvMeXbDvNmBiwfNjs5h963kPI7OYyqL9R2T7dynaf15Wvr7Z8x9lzwcWxHQhzf4dOT7DicBt9RyvADbJynJAiXIXfy6/AuYDPbPnF5CS3RYl4mbUvO+C3+Wg5v535W3d3NysZy3RG6T1gmaR1pX6HSkxFHowIgr7mIYB3YHfSaqs2YDJ2fl2yeJ2JDVnr8ny63sAUyPioca8mcyepPWNnikq3+OklWB3KngfYyNiUs0LI2IhaVbvNSLppGzU4wLSEvc15y61fH2pz6Urq9Zl2hN4FphQ9D4eIy3YuPWaltOskJv1rCU6gJRU5gPvRe2mJ6i9fHjf7HF0HeesWS7+4yy/vj5pAcY10Ze0Um/xIn2F54ZUvldLHF+jJcsljSAtp/4zUvPgbFJyHkvpJdXr+lxq+qj6AkNo+H2YfSxOTtYSvRqrRuvVpXhk3szs8WhWX1a9Rk1/U+Hy6+8WHM+z/PoMaq/smtdMUv9M8cq6NSZmj1PqKMuaLg9/GPD3iDijZoekwfXE1/W51CTlmcA0Sq/fBF6DyZqIk5O1Fs+QEtCQiPhtPXHPkjr4DwGuKNifZ/n1x4HDJO0TEXU1sy3NHtdjVUIEeJR039aCiHijnmuMAc6UtElEvA9phCBrvshhZ9KikIWOqSe+1OeygFW1uUdJCzNOiohpa1gmswY5OVmrEBHzJJ1JWqa9D/AIaYDEAGBX4KmIuCsixku6C6gZIl0zWu9rOS5zJ3A8cLeky0mJrhtptNq1WdJ5LYs9Q9IjQFVEPEfqNzuGtLT9T4GXgA7A5sC+wP4RsQi4BjiZdH/XSFaN1lu8hh/No8DZkn5IuqF5d+CgeuKPL/hchgPHASMjYk52/BrSSMd/ZsuvjycN2PgE8IWI2G8Ny2m2GicnazUi4iZJ75O+zL9BGmjwAfAP4MWC0BNJtYEfkBLEE1n8vxo4/3JJe5DuZTohe5xJGlo9Kwv7M2kQx8mkm4ZFGlW4XNJw4JzstYOBhcA7wF9II/SIiBmSvgT8nLTM+UzgRtL/1TWZJeMS0j1gp5P6mJ4mJZ1364jfjzQjxwWk5H4Zaeh7zWcwV9Lns7KcTUr+c0hJ6oE1KJ9ZSV6m3czMWhwPJTczsxbHycnMzFocJyczM2txnJzMzKzFcXIyM7MWx8nJzMxaHCcnMzNrcZyczMysxfn/aiddLxTt1IUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm_plot = plot_confusion_matrix(pipe_lr, X_valid_fold, y_valid_fold, display_labels=['Non fraud', 'Fraud']);\n",
    "cm_plot;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[48323,    10],\n",
       "       [   30,    54]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_matrix(y_valid_fold, pipe_lr.predict(X_valid_fold))\n",
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One could imagine doing this separately with 5 folds and averaging, but we're not going to bother."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Caution**: \n",
    "\n",
    "- scikit-learn's convention is to have true label as the rows, predicted as the columns. \n",
    "- Others do it the other way around, e.g. the [confusion matrix Wikipedia article](https://en.wikipedia.org/wiki/Confusion_matrix). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Let's understand confusion matrix\n",
    "\n",
    "Using the sklearn convention:\n",
    "\n",
    "|   X   | predict negative | predict positive |\n",
    "|------|----------|-------|\n",
    "| negative example | True negative (TN) | False positive (FP)|\n",
    "| positive example | False negative (FN) | True positive (TP) |\n",
    "\n",
    "- Perfect prediction has all values down the diagonal\n",
    "- Off diagonal entries can often tell us about what is being mis-predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Confusion matrix for multi-class: \n",
    "\n",
    "- We will talk about multi-class classification later.\n",
    "- But for now, FYI we can still compute a confusion matrix, e.g."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://scikit-learn.org/stable/_images/sphx_glr_plot_confusion_matrix_001.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision, recall, F1 score (10 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Accuracy does not tell the whole story!\n",
    "\n",
    "- Accuracy is misleading when we have class imbalance. \n",
    "- We need other metrics to evaluate our models. \n",
    "- We'll first discuss model evaluation.\n",
    "- And then what can be done about class imbalance.\n",
    "- We need to be proactive here and not blindly call `.score()` or `cross_val_score()` without thinking about metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Precision and recall\n",
    "\n",
    "Recall: how many of the actual positive examples did you identify?\n",
    "\n",
    "Precision: of the positive examples you identified, how many were real?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2XElEQVR4nO3dd5xcVf3/8dc7uymkJ5AEEggJBFAwgF9RiD8ERCUoXbpKlSIlIiJNWihKEQUVQhEFASkCiqBSjBQLCQpICxBaQggkpPe++/n9ce4mk9nZ3bthM7vZfT8fj/uYnXs/c++Z2WQ+e8o9RxGBmZlZS9KuuQtgZmZWzMnJzMxaHCcnMzNrcZyczMysxXFyMjOzFqeyuQtgH98GvSti0Cbtm7sY1ghvvty5uYtgjTSf2TMios+avn74F7vEzFlVuWKff3npYxGx55peqzVwcmoFBm3Snv88tklzF8MaYXj/7Zu7CNZIo+P+9z7O62fOquI/jw3MFVux0VsbfJxrtQZOTmZmZRBANdXNXYx1hpOTmVkZBMHyyNesZ05OZmZl45pTfk5OZmZlEARVni4uNycnM7MyqcbJKS8nJzOzMgigyskpNycnM7Mycc0pPycnM7MyCGC5+5xyc3IyMyuDINys1whOTmZm5RBQ5dyUm5OTmVkZpBkiLC8nJzOzshBVqLkLsc5wcjIzK4M0IMLJKS8nJzOzMkj3OTk55eXkZGZWJtWuOeXm5GRmVgauOTWOk5OZWRkEoop2zV2MdYaTk5lZmbhZLz8nJzOzMgjEsqho7mKsM5yczMzKIN2E62a9vJyczMzKxAMi8nNyMjMrgwhRFa455eXkZGZWJtWuOeXm5GRmVgZpQIS/cvPyJ2VmVgYeENE4Tk5mZmVS5fuccnNyMjMrA88Q0ThOTmZmZVLt0Xq5OTmZmZVBmvjVySkvJyczszIIxHJPX5Sbk5OZWRlE4JtwG8GflJlZWYjqnNsanV16VFJIuqxofy9Jt0iaIWmhpNGShpZ4fSdJP5E0RdJiSWMk7VIirp2kcyVNlLRE0kuSDqyjTMdLekPSUknjJX0n7/txcjIzK4Mg1ZzybI0l6XBguxL7BTwE7AmMAA4E2gNPStq4KPzXwPHAhcDewBTgMUnbF8VdCowErgO+CowF7pP0taJrHw/cBDyQXf8+YJSkk/K8JzfrmZmVydoYECGpJ3ANcDpwV9HhfYGdgd0j4sksfgwwATgL+G62bzvgG8CxEXFrtu9pYBxwSXYeJPUFfgBcERFXZ9d4UtIQ4Argr1lcJfAj4I6IOK8grj9wqaRbImJ5fe/LNSczszIIRHXk2xrpKmBcRNxd4ti+wIc1iQkgIuYCDwP7FcUtB+4tiFsB3AMMl9Qx2z0c6ADcWXSdO4GhkgZnz4cBfUrE3QGsT0qY9XJyMjMrgwCWR2WuLS9JOwNHAifXEbIN8GqJ/eOAgZK6FsRNiIhFJeI6AEMK4pYCb5eIA9i6II4S1y6Oq5Ob9czMykKNWc9pA0nPFTy/OSJuXu1sUntSn87VETG+jvP0BiaW2D8re+wFLMjiZtcT17vgcU5ERI44SpyzOK5OTk5mZmUQNGqGiBkRsUMDMWcD65H6duqi7NKl9q/tOOqIzcXJycysTJpqJVxJA4HzgOOAjgV9QmTPewLzSTWVUrWUXtljTc1mFjCwnrhZBY+9JKmo9lQqjuzaUwriehcdr5P7nMzMyiBCVEe7XFsOmwGdSAMOZhdskEbTzQaGkvp4tinx+q2BSRGxIHs+DhgsqXOJuGWs6mMaB3QENi8RB/BaQRwlrl0cVycnJzOzMkgDIipybTm8CHyxxAYpYX2RlFAeAgZI2rXmhZK6A/tkx2o8RLr/6eCCuErgUODxiFia7X6UlKy+WVSebwGvRsSE7PkYYEYdcbOAfzf0Bt2sZ2ZWFmqy6YsiYg7wVK0rSADvRcRT2fOHSIniTklnkmpU55L6hK4qON+Lku4Frs0GWkwATgIGU5BgImKapGuAcyXNB14gJbDdKRiaHhHLJV1Auun2A2B0FnMsMCIiljX0Hp2czMzKIA2IKO9igxFRLWlv4GpgFKkpcAzwxYh4vyj8GNLgisuAnsBLwJ4R8UJR3HmkEX6nARsC44FDIuLhomvfKCmAM4AzgUnAqRExKk/ZnZzMzMpkbS+ZEVE7+0XELFKN5dgGXrsY+H621RdXRUpgl9UXl8XeRBru3mhOTmZmZVAzQ4Tl4+RkZlYm1R6DlpuTkzWpH35jM55/qjuHnzaVo8+eunL/xPGduP2qDXn9hS4snNeOfpssY/hhszjguOlUZP8KP5rcnlHnb8y749ZjzsxKOq1XzaBPLOGQUz7is7vPX3muye905OHbNuClZ7oy5b0OdO5azZbbLeLIs6aw+TZLVivP1d/bhDde6MLMqe2proaNBi3jq4fPZO+jZ1Dhdd8abYONlnHIKdPYYtvFbLbNYjqtFxz5uU/y0eQOq8V17bGC4y6Ywuf3nEvHTsHrz3fmxov6M/GN9Zqp5M0vApZXOznl5eRkTebJP/Zkwmu1v3xmTq3krAOHsP6Gy/nOxR/Qo/cK/vevrtxyaX/mzKjkuPPTPXqLF1bQo/cKjjp7ChtstJxF89vxyF3rc/63NueCWyaw89fmAvD809146ZmufPngWQwZupiF8yq4b1RfTtt7S67501tsse3ilddetqQd+x07nY0GLUPAc09344YLB/DhxI6cdOkHZflcWpP+g5axyz5zeevl9Xj12S7ssNuCElHBxbdNpN8myxh1/gAWzKng0BHTuOr+dzj5K1syY0qHEq9p/VKznpNTXmVNTpKOBm4F5gKDI2J2wbFK0qy4F0fEyDKXaxhwLfApoDPw6Yh4sZxlKCUb6VL2z2NNLJhbwU0jB3DiyA+44pRBqx17dnQP5s6q5Gd/eouNN0+3S2y/8wKmTOzI3+/vvTI5DdpqCd//2eoDiHb88jyO3HFrHr+n98rktNt+s9n3mBmooPl++/83nyN33Jo/3tKHs34xaeX+H9743mrn+8xu85k1tT2P3dPbyWkNvDK2C4dtl+6r3PMbM0smp2HD5/GpHRdy1kGb89IzaV7R157vwu1jX+fgk6dzwwUDylrmlqSpZohoC5orjfcgzQvVUvyalKj3IU31/mbzFmfdc8tlG7HpVkv44gFzah1bviz9h+zcrWq1/V16VFFdXf95KyqhS/cqKtuvmimlx/pVqyUmgC7dqxmw2VJmTm3fYFm79VpBReUaT/nVppUYDFbLTnvMY8aUypWJCWDR/ArG/q07w4bPXZvFa9FqhpKvhSUzWqXmSk6PAyMkbdhM119JUjtgK+AvEfFERIwtMW18TWzHUvvbulef7cLo+3tz6o8nlzy+yz5z6NF7BdeftzFTJ3Vg4fx2/PuRHvz9/l4ceOL0WvHV1VC1AmZNq+R31/Tjg3c7ss/RM+otw7zZFUx8oxObDFlS61hEOt+CuRX88y89GH1fb75+Qu3rWtPYdMslTBzfqdb+997sRL+Nl9Opc1WJV7UFTTp9UavXXJ9Czfj48+qNAiR9LlvzfoGkhZL+LulzRTG3SZos6dOS/ilpkaS3GlqvPmtmrCJ9DhdICkkTi845TNIzkhaT3VEt6TBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUml5sNqcVYsh5+fvTEHfWcamwxZWjKmV58VXPPwm0x6qyNH7bQ1X99qWy49bhCHnDKNQ06ZViv+lkv787WB23P49p/ivlF9OfeG9/j0F0r1bawy6vwBEOKA42snnWdHd+drA7fnwE8O5UcnDGLfY2fwzdM/WrM3bA3q1nMFC+bW7jGYP6ciO95WkxNUo1ybNd+AiCmk9ee/J+nqiHivVJCkbYGnSZMEHk2qGZ8DPC1pp4h4qSC8O2mJ4mtJywofA9wgaXzhKpBF/kJakfFfpKa9W0gLadXoQVoJ8mrgh0BNT/tmwP2kZYmrgV2AWyStFxE35vwMCo3Mzv8zUq1yB1af96rF+v31/Vi2pB2Hf7fuL/s5Myu49LjBdOpczfm/mkD3XlW8+K+u3P3zfrTvEBx66uoJ6oDjp7Pb/rOZNa09o+/vxRWnbMr5N09kp6/MK3n+e37Zlyf/2Jvv/3QSAwbXnhVl6I4L+OUj41k4r4L//asbD9zYByk45pypJc5mH5eUaqu19pe/KC1KGq3nIaJ5NedovSuBE4GLqPvO5QtJyeJL2VxSSPobafGsi4CvF8R2A06uSUSS/gHsARwOlExOETFdUs2gjMkRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEWkuqkYlJ0m9gNNJi4n9INv9uKQqUvKr63UnACcADBzQPL/GaZPbc/cv+nH61e+zfFk7lhfkheVL27FgbgXrda3ivuv78dH7Hbj9P6+t/Kt5u88voLoabv/Jhux5+Ex6rL/qr+k+/ZfTp/9yYDE7fWUeZx44hF9d0r9kcvrz7etz6+X9OersKQw/vPQs/F26V7Pldunvik9/YQHt2wd3XduPfY6ayQYbLW+6D8QAmD+nkm49V9Ta3zX73dfUoNoa34TbOM3WuJlNqfFT4EhJW9URtgvw55rElL1uHqlWsWtR7KLCGlI2i+5blF6jJK8VwJ+Ld0raQtLd2YSGy7PtOFLfVWMNBboAvy/af099L4qImyNih4jYoc/6zfOffcqkjixb0o4rT92UAz85dOUGcP+NfTnwk0OZ+Pp6THijE/0HLa3VnLPV9otYsbwdH06svytvi20XlYwZfX8vrjt3Yw48cRrfOC1/M90W2y2iulpMndQ2hzSvbe+92ZFNt6zdxDtwiyV8NLk9Sxa1zeQEbtZrjOa+z+kaYASpGa54anWovVBVjamsWtyqRqklhpeSJjpcU9OyeaRWktQV+BuwiNTE+A5pCvmTaGDuqjpslD0Wf7u2+E6RzbdZzFX3v11r/1kHDeFLB85i+OGz6D94Kb37ruD157owf07Fagnqjf91AWD9DeuuvVRXw7j/dqH/pqt/2f37kR789PSB7PmNmZxw0YeNKvcrY7oiBRttWrqPzD6eMY/3YPhhsxm60wJeGZtG7HXuWsVOX5nHkw/2bN7CNaPmmPh1XdasySkiFki6nFSD+kmJkFmkWW+LbUiOlRSbQKnxxsOATYEvRMS/anZm92kVqhk2Vvzn+fpFz2uSbz9WLdBV87xF69qjiu0+X3qgQt+Nl608ttcRM3jiD7344eGbc9BJ0+jeawUvj+nKAzf24f99dQ59B6TkdMfVGzJ/TgVbf3YhvfsuZ9a09jx2d2/G/68z51y/qlvylbFduPzkTdnsk4v5yiGzeP35Veujte8QDBmamvCeHd2dx+/tzU5fmUufActZvKAd/32yO4/cuT5fO2Im629Yu+nJGrbzXnMA2CL7nD+7+zzmzKxk7sxKXhnblbGPdee15zpz9nWT+NWlG628CVeC+0b1bcaSNz+PxMuvuWtOkKZx/z6lZ7h9GthLUreImA8gqRvpfqSnylbC1dV8E678cz/rN9qvKO4jUs3tU0X79yp6/jKwEDgEeKJg/2Efu6QtxCc/s4if/vEtfnfNhtx44QAWzk/TF33z9I848MRVgyGGDF3EH2/pw1N/6smi+RX06rOCzbZezE//+DbbfG7hyrgX/9WN5Uvb8farnfn+fluudq1+Gy/j9v+kRTY32nQpUQ23XbkRc2dW0qV7FQMGL+XMX0xit/1LVbQtjwt+tfr4pRFXpJuZX3qmC2cdNIQIceGRgzn+wg8Z8eMPaN+pmtef68JZB2/O9A/bblNqhFjh5JRbsyeniFgq6RLg5hKHLwX2Bv4u6UpSTeZsUoK4pHylXM0zwDzgekkXkfqLziet+tijJigiIlu869uS3iStebIXsFvhySJiTrZ413nZ4l2PA58Fvl2G97JWPPbhi7X2ffIzi7jsznfrfd2w4fMYNrz0iLxCR/xgKkf8oOGRdgO3WMqFv57YYJw1zvD+2zUYM39OJT/7/kB+VobyrEvcrJdfS0njt5IGL6wmIl4mfZnPA34L3EFa5GrXomHkZRMR04EDgArScPLLSUPQ7ywRfhrwB9JQ8XtJ/V8jSsSNBH4MHEEa7LEHqXZoZq2EZ4honLLWnCLiNuC2EvtXAFsW78+OPQt8uYHzHl3H/t1ylGkFJW7BqOuc2bEngE+XODSyKG4OKeEUU1FcFan2dX59cWa2bnPiya/Zm/XMzNoC3+fUOE5OZmZl4nuY8nNyMjMrgwhY4cUGc3NyMjMrEzfr5efkZGZWBu5zahwnJzOzMsmzWKMlTk5mZmXiARH5OTmZmZVBhPucGsPJycysLESVR+vl5uRkZlYm7nPKr87kJKma0ktGlBIR4URnZlYHr+fUOPUllEvIn5zMzKw+kfqdLJ86k1NEjCxjOczMWj2P1suv0U1x2TLl6wMfRkTd62ubmdlK4QERjZL7k5K0t6QXgLnAu8DQbP8tkr6xlspnZtZqROTbLGdykrQ/8CfSaq9ns/o6QxOAo5q8ZGZmrUyEcm0NkTRc0hOSpkpaKmmypN9L2roorldWgZghaaGk0ZKGljhfJ0k/kTRF0mJJYyTtUiKunaRzJU2UtETSS5IOrKOMx0t6IyvfeEnfacRHlbvmdBFwa0TsAVxbdOxV4FONuaiZWVuTakVNk5yA3sDzwKmklbPPBbYBxkraFECSSCtr70lagftAoD3wpKSNi873a+B44EJgb2AK8Jik7YviLiUtqnod8FVgLHCfpK8VBkk6HrgJeCC7/n3AKEkn5XlzkL/P6ZPAWdnPxZXO2aQ+KDMzq0dTDSWPiLuBuwv3SfoP8AZwEPBTYF9gZ2D3iHgyixlDau06C/hutm874BvAsRFxa7bvaWAcadT2vtm+vsAPgCsi4urssk9KGgJcAfw1i6sEfgTcERHnFcT1By6VdEue8Qp5a07zgA3qODYImJ7zPGZmbdZa7nOamT3WfPHvSxq49uSq68dc4GFgv4LX7Zu95t6CuBXAPcBwSR2z3cOBDsCdRde9ExgqaXD2fBjQp0TcHaSKzM553kze5PQ34FxJPQv2RVboU4FHcp7HzKxNCkR1dbtcW16SKiR1kLQFqRltKimpQGrme7XEy8YBA7OR1zVxEyJiUYm4DsCQgrilwNsl4gC2LoijxLWL4+qVt1nvPOA/wHhS1S2Ac4BtgR7A/jnPY2bWZjWiUrSBpOcKnt8cETeXiHsW+Ez289ukJrxp2fPewMQSr5mVPfYCFmRxs+uJ613wOCeiVt2uVBwlzlkcV69cySkiJkr6P+BiUtWuCtgFeBS4MCI+zHMeM7M2Kxo1t96MiNghR9wRQHdgM1J/0N8k7RwRE0mjqkvlw+JCrI046ojNLfdNuBExGfj2x7mYmVmb1sT3MEXE69mPz0p6hFRTOgf4DqmmUqqW0it7rKnZzAIG1hM3q+CxlyQV1Z5KxZFde0pBXO+i4/Vq9O3KkvpL+mw28sLMzHJqwqHkJc4dc0hNezV9RONY1f9TaGtgUkQsKIgbLKlzibhlrOpjGgd0BDYvEQfwWkEcJa5dHFevxswQcaSkCcD7pLHt70uaIOlbec9hZtZWBVBdrVzbmpDUD/gE8E626yFggKRdC2K6A/tkxyiIaw8cXBBXCRwKPB4RS7Pdj5KS1TeLLv0t4NWImJA9H0OasKFU3Czg33neT65mPUmnAr8ARpNuwvoI6AccDvxWUo+IuD7PuczM2qQAmug+J0l/BF4AXibd6rMlcDqwgnSPE6SkMwa4U9KZpGa8c0l9QletLFbEi5LuBa6V1J50H9RJwGAKEkxETJN0DWnk9vzs+ocCu1MwND0ilku6gHTT7QekvLE7cCwwIiKW5XmPefuczgBui4hji/b/RtJtpI44Jyczs3o04bx5Y4FDSN/NHUgtWk8Bl2eDIYiIakl7A1cDo4BOpGT1xYh4v+h8x5BunL0M6Am8BOwZES8UxZ1HGuF3GrAhaQT3IRHxcGFQRNwoKbLynQlMAk6NiFF532De5LQhq8bOF7uL9CGZmVl9mig5RcSVwJU54maRaizFFYviuMXA97OtvrgqUgK7LMe1byLde7VG8ianV6jdCVZjC0rf6GVmZiut+WCHtihvcjoNuEfSDOAPEVElqYI0keCZwGFrq4BmZq2Gl8PIrc7kJOl9Vv8oe5Ca9qokzSaNba8gtT/eC2y6FstpZrZuC4g1HInXFtVXc/o7zvNmZk3IySmvOpNTRBxdxnKYmbV+/nM/t9zTF5mZ2cfk5JRbo5JTtijVVqTx8quJiNubqlBmZq1OE96E2xbknSGiJ/AXYKeaXdlj4d8BTk5mZvVowptwW728c+v9mLSC4S6kxHQAaTqK3wHvAp9bK6UzM2tNqpVvs9zJaTgpQY3Nnk+OiKci4kjSvEmnrY3CmZm1Jop8m+VPThsB72ZTVywBuhUc+wOwV1MXzMysVYlGbJY7OU0lTQYI8B4wrODYkFrRZmZWRGlARJ7Nco/W+xcpIf0ZuAO4SNIg0vTsR7H62iBmZlaKa0W55U1OFwM1K9/+hDQ44lCgMykxjWj6opmZtTLVzV2AdUeu5BQR75CtrhgRy0lrdJyxFstlZta6+D6nRvEMEWZmZeKRePnVNyv5hY04T0TEpU1QHjOz1svJKbf6ak4jG3GeAJyczMysSdQ3K3neYebWzN58uTPD+2/f3MUwswa4WS8/9zmZmZVD4KmJGsHJycysXFxzys3JycysTNysl5+Tk5lZuTg55ebkZGZWLk5OuTk5mZmVgZfDaJzGLtO+LWnBwfWBmyJiqqQhwEcRMX9tFNDMrNXwaL3c8i7T3hG4E/g6aSXcAB4mLaVxFfAmcM5aKqOZWavgmlN+eW+0/RHwZeAIoB8pQdV4hLRSrpmZ1ceLDeaWt1nvcOD8iLhLUkXRsQnAoCYtlZlZa+M+p0bJm5zWB16v41g7oGPTFMfMrBVzcsotb7PeBFZfmr3Q54DxTVMcM7PWS9X5NsufnG4HzpH0TaBDti8kfRE4HfjN2iicmZm1TXmT01XAX4A7gFnZvn8Bo4FHI+KXa6FsZmatSxMNiJB0kKQHJL0nabGk8ZIul9StKK6XpFskzZC0UNJoSUNLnK+TpJ9ImpKdb4ykXUrEtZN0rqSJkpZIeknSgXWU8XhJb0hampXvOw2/s1VyJaeIqIqIw4BdgZ8CtwC/AHaPiG825oJmZm1SrLoRt6Ethx8AVcAPgT2BG4CTgL9JagcgScBD2fERwIFAe+BJSRsXne/XwPHAhcDewBTgMUnbF8VdSlrr7zrgq8BY4D5JXysMknQ8cBPwQHb9+4BRkk7K9e5o5E24EfFP4J+NeY2ZmWWabkDEPhExveD505JmAb8FdgOeAPYFdiZVIp4EkDSGNIbgLOC72b7tgG8Ax0bErdm+p4FxwCXZeZDUl5QUr4iIq7PrPplNxHAF8NcsrpJ0+9EdEXFeQVx/4FJJt0TE8obeoBcUNDMrlyZq1itKTDX+mz0OyB73BT6sSUzZ6+aSJlDYr+B1+wLLgXsL4lYA9wDDs0kYIN3P2oE0IUOhO4GhkgZnz4cBfUrE3UEa+b1zQ+8PciYnSdWSqurb8pzHzKytEmt9tN6u2WPNbT/bAK+WiBsHDJTUtSBuQkQsKhHXARhSELcUeLtEHMDWBXGUuHZxXL3yNutdQu18vj6wB+kep9tynsfMrG1q3E24G0h6ruD5zRFxc13BkgaQvqdHR0TN63oDE0uE1wxq6wUsyOJm1xPXu+BxTkQUv4tScZQ4Z3FcvXIlp4gYWWp/NlvEw8DcPOcxM2vT8ienGRGxQ57ArAb0J2AFcEzhoTquWDz77NqIo47Y3D5Wn1NEVAGjgO99nPOYmbUJTTy3nqROpBF5mwHDI2JyweFZlK6l9MoeZ+eMm1Xw2CsbBdhQHCXO2bvoeL2aYkBExxKFMDOzIk04lBxJ7UlDtT8HfC0iXikKGceq/p9CWwOTImJBQdxgSZ1LxC1jVR/TONL3/eYl4gBeK4ijxLWL4+qVd0DEwBLbEEn7k4YQPtfAKczMrOluwm0H/A74ErBfRIwtEfYQMEDSrgWv6w7skx0rjGsPHFwQVwkcCjweEUuz3Y+SklXxva3fAl6NiAnZ8zHAjDriZgH/bvgd5h8QMZG62xrfAU7JeR4zs7YpmnTevOtJyeRHwEJJOxUcm5w17z1EShR3SjqT1Ix3Lul7+6qVxYp4UdK9wLVZbWwC6YbewRQkmIiYJuka4FxJ84EXSAlsdwqGpkfEckkXkG66/YA0k9DuwLHAiIhYlucN5k1Ox5TYtwR4D/hv1vdkZmb1abqbcL+aPZ6XbYUuBkZGRLWkvYGrSWMDOpGS1Rcj4v2i1xxDSnSXAT2Bl4A9I+KForjzSCP8TgM2JE36fUhEPFwYFBE3SgrgDOBMYBJwakSMyvsGG0xO2Yi8F0k3c5W68cvMzHJoqvWcImJQzrhZpBrLsQ3ELQa+n231xVWREthlOa59E2kKozWSp88pSH1Kn17Ti5iZGV4JtxEarDllVcP3gS5lKI+ZWevkxNMoeYeS3wR8T1KHBiPNzKwW0bRDyVu7vAMiupHGtr8r6VHSdOqFH2FExEVNXTgzs9bEiSe/OpOTpHeBAyLiJdKaITVKdawF4ORkZlYfJ6fc6qs5DSLdDUxEeGkNM7OPy8kpt0YtNmhmZmvI/UmN0lBy8kdpZtZU/I2aW0PJ6WJJM3KcJyLiqKYokJlZa9WE0xe1eg0lp+1JKx82xH8PmJk1wM16+TWUnPaPiP+UpSRmZq2Zb8JtFA+IMDMrFyen3JyczMzKoGaGCMvHycnMrExU7eyUV53JyTfempk1Ifc5NYprTmZmZeJmvfycnMzMysXJKTcnJzOzMnHNKT8nJzOzcnFyys3JycysHMLTFzWGk5OZWRn4PqfGcXKyZvOZXedxyCnTGbjlErr2qGLuzEpef64zd/x0Qya91WllXNceKzjugil8fs+5dOwUvP58Z268qD8T31ivGUvf9mw7bAE/eeCdWvsXzG3HgZ8cWvI1373yffY6YhZ/f6AnV43YdG0XseULZ6e82nRyknQ0cGsdh78SEaPLWJxaJD0FEBG7NWc51pZuvap465X1ePi36zN3ZiV9Byzj0FOnce2f3+I7u2/FtA86AMHFt02k3ybLGHX+ABbMqeDQEdO46v53OPkrWzJjSofmfhttzvXn9+fNFzuvfF61QiXjtt5hIbt/fQ4L5/mWyRquOeXXppNTgYOByUX7XmuOgrQlTz3Yi6ce7LXavvEvdubX/xzPF/aewwM39WXY8Hl8aseFnHXQ5rz0TFcAXnu+C7ePfZ2DT57ODRcMaI6it2nvv9WJN17oUm9MRWVw2lWTufvnffnaETPLVLIWzjfhNor/pElejIixRdu84iBJHZujcG3JvNnp76UV2V/jO+0xjxlTKlcmJoBF8ysY+7fuDBs+t1nKaA07+KRptKsIHripT3MXpUVRdb7NnJzqJOloSSFpF0n3SZoDPJsd+6yk+yVNlrRY0nhJP5a0XtE5Jkq6rcS5Q9LIon2HSXpD0lJJ4yQdsPbeXcvSrl1Q2b6a/oOXctqVk5n5USVPPdgTgE23XMLE8Z1qvea9NzvRb+PldOpcVebS2tnXTeKv77/Efa++yjnXv0efActWO77Rpks5/Hsfcd25G7Niub9iCjk55edmvaRCUuFnUVj5/h1wN3AQqz6vgcCLwG3AfGAb4EJgM+Cwxl5c0peBu4C/AGcAfYCfA+2B8Y0937rm5395iy23WwzAB+924OyDN2fuzPYAdOu5go8md671mvlzKrLjVSxZVFG+wrZhC+e34/4b+/DymC4sml/BkE8t5rDvTuPah97i5D22XPk7++6Vk/n3X3usVts1smY9t+vl5eSUvFH0/N/ALdnP90fEWYUHI+KBmp8lKYufB9wu6ZSIaGwj+8VZGfaLiOrsvK8DY6kjOUk6ATgBoBO1v7zXJVeNGEjnblVstOkyDvrOdC6/513O2H8IH03ugFT6/3PpLnhbm955tTPvvLrq39orY7vyyrNd+MVf3mL/b8/gt1dtxO5fn82W2y3iuF0+0Ywlbbk8ICI/17mTA4DPFmzfLjj2x+JgSd0lXSnpHdIy9suBO0jfmVs05sKSKrJr3l+TmAAi4llgYl2vi4ibI2KHiNihPet2V9j7b3di/P+68NSDvTj7kM1Zr0sVh546DYD5cyrp1nNFrdd07VmVHXetqTm9/UpnJr/bkS23X0SnzlWcOPJDfn99X5YtbUeX7lV06V5Fu3ZQ2T7o0r2Kiso2/u0cOTdzzSnzakS8XbhD0rDsxykl4m8FvkxqynsRWAh8DrgeqN1BUr8NSM13H5U4Vmpfq7ZwXgUfTuxI/8FLAXjvzY783y4LasUN3GIJH01u7ya9FkACQvToXUXPDVZw7A+ncuwPp64W03fAXHbddy4jjx3EmEd7NE9Bm5lvwm0cJ6eGrfbPSVInYD9gZET8vGB/qbsQlwCr3YgjqXdRzAxSzatfidf3A95bgzKvs3pusJxNhizliT/0BGDM4z0Yfthshu60gFfGpj6Mzl2r2Okr83gyGzRhzWeLbRcxYLOl/OPhnsyaXsmZB25eK+bcG95j4huduPvn/UoObmkzIrzYYCM4OTVeR6CClFAKHV0i9j3gU0X79i58EhFVkv4LHCRpZEGf047AIFpxcrrw1xN4+5XOTHi9E4vmVzBgs6V8/YTpVK2AB27qC8DYx7rz2nOdOfu6Sfzq0o1W3oQrwX2j+jbzO2hbzr7uPaZO6sDbr6zHgnlpQMShI6Yxc2p7/vSbDVi+tB0vj6k9CGLZUjF7emXJY22Oc1NuTk6NFBFzJY0FzpA0hVTzORYodTfoPcBvJF0D/BnYjtJJ7CLgceBBSTeRRutdDEwtEdtqvPFCF3bZZw4HnjiNyg7B9A878PIzXbj3l/34aHKqcEaIC48czPEXfsiIH39A+07VvP5cF846eHOmf+jZIcpp4hud2G3/Oex37Aw6rlfN7Ont+fdfe3DH1Rsyb5a/SvJoymY9SRsDZwM7kL5b1gMGR8TEorhewE+A/bOYMcDpEfFKUVwn4FLgW0BPUpfF2RHxj6K4dtl1TwQ2JA3auqRwoFhB7PGkEciDSX3o10TEjXnen/9FrZnDgRtIfUyLgd8Dp5ESUKHfApuQBlicCPyTNPhitf6tiBgt6ZvASOAP2fHvZedstX5/fV9+f33DtZ/5cyr52fcH8rMylMnqdu91/bj3ulKtz/U7aset10Jp1kEBNG2z3hDgEOB50nfLHsUB2Wjih0jJYQQwGzgXeFLS9hFRODPOr4G9gDOBd4FTgMckDYuIFwviLgV+AJyXXfsw4D5Je0fEXwuufTxwE3A5MBr4EjBKkiLihobenMLj7td53dU7dtSXmrsYZq3a6Lj/+YjYYU1f363HxvF//++7uWL/8cjZDV5LUruCboDjgF9RVHOStB/wILB7RDyZ7esBTADujIjvZvu2I9WUjo2IW7N9lcA4YHxE7Jvt6wu8D1wRERcVXOfvQJ+I2LbgtR8Cj0TEUQVxvwH2BTaKiOKukdV4KLmZWZko8m15FN56Uo99gQ9rElP2urnAw6SBXYVxy4F7C+JWkLomhhdM3TacNMjrzqLr3AkMlTQ4ez6M1D1RHHcHsD6wc0MFd3IyMysTVUeurQltA7xaYv84YKCkrgVxEyJiUYm4DqQmxJq4pRR1TWRxAFsXxFHi2sVxdXJyMjMrh7w34KbctIGk5wq2E9bwqr1J/UzFZmWPvXLG9S54nBO1+4NKxVHinMVxdfKACDOzMkg34eauFc34OP1bRZctddHiGcDWRhx1xObimpOZWblU59yazixK11Jqakyzc8bNKnjslY0CbCiOEufsXXS8Tk5OZmZloohcWxMax6r+n0JbA5MiYkFB3GBJxbNIbw0sY1Uf0zjSRATFU4HU9CG9VhBHiWsXx9XJycnMrBwa1+fUVB4CBkjatWaHpO7APtmxwrj2pFXBa+IqgUOBxyNiabb7UVKy+mbRdb5FmqN0QvZ8DGmCglJxs0grOdTLfU5mZmXR9HPrSToo+/Ez2eNXJU0HpkfE06SkMwa4U9KZrLoJV8BVK0sW8aKke4FrJbUn3Qd1Eunm3W8WxE3LZrw5V9J84AVSAtudgqHpEbFc0gWkm24/IN2EuztpNp0REbH6CpUlODmZmZVL0096cF/R81HZ49PAbhFRLWlv4OrsWCdSsvpiRLxf9NpjgB8Bl5GmL3oJ2DMiXiiKOw9YQJrBpmb6okMi4uHCoIi4UVKQpi86E5gEnBoRo8jBycnMrByi6Zdgj4gG192MiFmkGsuxDcQtBr6fbfXFVZES2GU5rn0TaQqjRnNyMjMrF08Xl5uTk5lZuTg35ebkZGZWJqpu4na9VszJycysHIKmvsG2VXNyMjMrA9HkN9i2ak5OZmbl4uSUm5OTmVm5ODnl5uRkZlYO7nNqFCcnM7My8Wi9/JyczMzKItys1whOTmZm5RA4OTWCk5OZWbm4VS83JyczszLxfU75OTmZmZWLk1NuTk5mZuUQAVVu18vLycnMrFxcc8rNycnMrFycnHJzcjIzK4cAqp2c8nJyMjMri4Bwn1NeTk5mZuUQeEBEIzg5mZmVi/uccnNyMjMrFyen3JyczMzKwhO/NoaTk5lZOQTgJTNyc3IyMysX15xyc3IyMysLT1/UGE5OZmblEBC+zyk3Jyczs3LxDBG5OTmZmZWL+5xyc3IyMyuHCI/WawQnJzOzcnHNKTcnJzOzsgiiqqq5C7HOcHIyMysHL5nRKE5OZmbl4qHkuTk5mZmVQQDhmlNuTk5mZuUQXmywMZyczMzKxAMi8lN4aOM6T9J04L3mLsdasgEwo7kLYbm15t/XphHRZ01fLOlR0ueTx4yI2HNNr9UaODlZiybpuYjYobnLYfn492VNpV1zF8DMzKyYk5OZmbU4Tk7W0t3c3AWwRvHvy5qE+5zMzKzFcc3JzMxaHCcnMzNrcZycDElHSwpJcyT1KjpWmR0b2QzlGibpWUkLszJsX+4ylNJcn0dTK/i9l9q+3ALK95Skp5q7HNY8PEOEFeoBnA2c09wFyfwaWAzsAywC3mze4rRaBwOTi/a91hwFMavh5GSFHgdGSLo2IqY2Z0EktQO2An4UEU80ENsxIpaWp2St0osR8XZDQf6crZzcrGeFLssez2soUNLnJI2WtCBrdvu7pM8VxdwmabKkT0v6p6RFkt6S9J0Gzn00UEX693lB1sw0seicwyQ9I2kxcFV27DBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUnbNPTZtAYFTX67SLpP0hzg2ezYZyXdn/0eFksaL+nHktYrOsdESbeVOHetZtHsd/eGpKWSxkk6YO29O1sXODlZoSnAdcAJkjatK0jStsDTQC/gaOBIoDvwtKTtisK7A3cBdwL7Af8FbpD0xXrK8Rdg5+znXwPDgMIvqx7APcDdwFez8wNsBtwPfBPYH3gYuKWhZFiPkcAPgd9l53sceGgNz9WSVWR9izVbRcGx3wETgINY1dw7EHgR+A6wJ/Bz4Fjg1jW5eNa/dRfwFvB14CfZObdak/NZ6+BmPSt2JXAicBHpC6eUC4GlwJciYg6ApL8BE7PXfb0gthtwckQ8mcX9A9gDOBx4stTJI2K6pNnZ08kRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEXAScGNdb7iUbGDI6cDNEfGDbPfjkqqAKxpzrnXAG0XP/w3ckv18f0ScVXgwIh6o+VmSsvh5wO2STomImY28/sVZGfaLSGtKSHodGAuMb+S5rJVwzclWExGzgJ8CR0qq6y/XXYA/1ySm7HXzSLWKXYtiF9UkpixuKekv5IEfo5grgD8X75S0haS7JX0ALM+241izv8CHAl2A3xftv2cNztXSHQB8tmD7dsGxPxYHS+ou6UpJ75D+SFkO3AEI2KIxF85qaZ8lJcGVix1FxLOkP3asjXJyslKuAWYBl9RxvDepCbDYVFJTX6HZJeKWAp3WuHQwLSJWWxhHUlfgb8B2pOanL5C+9H4DdFyDa2yUPX5UtL/4eWvwakQ8V7AV1lZK/Z5vJTXp/QL4CulzPiU71tjf6wZAe0p/rq3xs7ac3KxntUTEAkmXk2pQPykRMgvYsMT+DbNja1upObeGAZsCX4iIf9XslFT8b3xJ9tihaP/6Rc9rvpT7AeMK9vdrXFHXeat91pI6kfoOR0bEzwv2Dy3x2iUUfc6SehfFzCDVvEp9rv1oveuUWQNcc7K6jAI+YNUIvkJPA3tJ6lazI/t5n+xYc+icPS6v2ZH1G+1XFPcRqeb2qaL9exU9fxlYCBxStP+wj1fMdV5HoIKCzzlzdInY96j9Oe9d+CSrAf8XOCjrJwRA0o7AoI9ZVluHueZkJUXEUkmXUHqW6UtJXzJ/l3Ql6a/rs0kJoq6mwLXtGVKn/PWSLiL1F51P+su8R01QRISke4FvS3qT1OG+F7Bb4ckiYo6ka4DzJM0njdQr7o9pcyJirqSxwBmSppA+32OBASXC7wF+k32OfyY1uR5dIu4i0uf7oKSbgD6kQRLNeq+dNS/XnKw+t5IGL6wmIl4mfZnPA35L6gxfAOwaES+Vs4AFZZpO6tivIA0nv5w04uzOEuGnAX8gDRW/l9RPMqJE3Ejgx8ARpMEee5Bqh23d4cDzwPXAbaQkclqJuN+yavTmw8BwVr8lAICIGE0a/r8V6fdyJvA9PFKvTfOSGWZm1uK45mRmZi2Ok5OZmbU4Tk5mZtbiODmZmVmL4+RkZmYtjpOTmZm1OE5O1mKUWDZ8vqSXJJ1aYhqipr52rXWesrWjJjbyPLtJGlk420ETlW+kpAbv+6hrDaW852+qz7ngdzmoKc5nbY+Tk7VEB5PmyjsQ+A/wS9IyHeV2KSVuGm3AbqQbT/1/y+xj8PRF1hIVLhv+uKQhpBkDSiYoSe2BFdHEd5RHxDtNeT4zy89/3dm64L9AN0l9C5rfTpZ0laQPSRO59gSQ9HVJY7Nl1edkS4yvtnaUpM6SRkmaqbSc+0PAxsUXLdWsJ6mLpCskvZMtKT5V0gOS+mVLj1+UhS6vaZ4suu6VkiZIWpY9nlfcBKhVy9ovkfSBpAtIayU1mqQ+km6S9Gb2mbwv6S5JpebCA/ikpCez2CmSLilRvg0k3ZCVbanS8uonrEn5zOrimpOtCwYDVaT5+2pmHz+PlLROIM2nt0RpOfYbSHMCXkJahXckafn4bSNifvbam4BDSZOL/pe0JlHNUu91ktSBtGbU9qS5+8aSJpUdTlrH6hZSkvs2aZn5qoLXVgKPAVuTmgtfAXYCLiCtj3VGFrcB8ARpvrqjSIn3TNZ8ccbepKUrzgWmA/2za/1b0iciYklR/IOkNbAuz97XBUA16XNEUnfSyrfrZfsmZHE3SOoYEb9cw3KarS4ivHlrERtpxuogTQBaSfrCP5H0Jf9gFjMoi3mBbG7IbH9XYC7wm6JzDgKWAd/Lnm+Vne+corgbsvMeXbDvNmBiwfNjs5h963kPI7OYyqL9R2T7dynaf15Wvr7Z8x9lzwcWxHQhzf4dOT7DicBt9RyvADbJynJAiXIXfy6/AuYDPbPnF5CS3RYl4mbUvO+C3+Wg5v535W3d3NysZy3RG6T1gmaR1pX6HSkxFHowIgr7mIYB3YHfSaqs2YDJ2fl2yeJ2JDVnr8ny63sAUyPioca8mcyepPWNnikq3+OklWB3KngfYyNiUs0LI2IhaVbvNSLppGzU4wLSEvc15y61fH2pz6Urq9Zl2hN4FphQ9D4eIy3YuPWaltOskJv1rCU6gJRU5gPvRe2mJ6i9fHjf7HF0HeesWS7+4yy/vj5pAcY10Ze0Um/xIn2F54ZUvldLHF+jJcsljSAtp/4zUvPgbFJyHkvpJdXr+lxq+qj6AkNo+H2YfSxOTtYSvRqrRuvVpXhk3szs8WhWX1a9Rk1/U+Hy6+8WHM+z/PoMaq/smtdMUv9M8cq6NSZmj1PqKMuaLg9/GPD3iDijZoekwfXE1/W51CTlmcA0Sq/fBF6DyZqIk5O1Fs+QEtCQiPhtPXHPkjr4DwGuKNifZ/n1x4HDJO0TEXU1sy3NHtdjVUIEeJR039aCiHijnmuMAc6UtElEvA9phCBrvshhZ9KikIWOqSe+1OeygFW1uUdJCzNOiohpa1gmswY5OVmrEBHzJJ1JWqa9D/AIaYDEAGBX4KmIuCsixku6C6gZIl0zWu9rOS5zJ3A8cLeky0mJrhtptNq1WdJ5LYs9Q9IjQFVEPEfqNzuGtLT9T4GXgA7A5sC+wP4RsQi4BjiZdH/XSFaN1lu8hh/No8DZkn5IuqF5d+CgeuKPL/hchgPHASMjYk52/BrSSMd/ZsuvjycN2PgE8IWI2G8Ny2m2GicnazUi4iZJ75O+zL9BGmjwAfAP4MWC0BNJtYEfkBLEE1n8vxo4/3JJe5DuZTohe5xJGlo9Kwv7M2kQx8mkm4ZFGlW4XNJw4JzstYOBhcA7wF9II/SIiBmSvgT8nLTM+UzgRtL/1TWZJeMS0j1gp5P6mJ4mJZ1364jfjzQjxwWk5H4Zaeh7zWcwV9Lns7KcTUr+c0hJ6oE1KJ9ZSV6m3czMWhwPJTczsxbHycnMzFocJyczM2txnJzMzKzFcXIyM7MWx8nJzMxaHCcnMzNrcZyczMysxfn/aiddLxTt1IUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm_plot.plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall = TP/(TP + FN) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6428571428571429"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "54/(30+54)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "^ That is how many of the frauds we caught."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision = TP/(TP + FP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84375"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "54/(10+54)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "^ Of the frauds we \"caught\", that many were actual frauds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [MDS terminology document](https://ubc-mds.github.io/resources_pages/terminology/) for related terms like sensitivity, specificity, Type I error, Type II error, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which metric do we care about?\n",
    "\n",
    "- Of course we'd like to have high precision and recall (more on this later).\n",
    "- But the balance depends on our domain.\n",
    "- For credit card fraud detection, recall is really important (catching frauds), precision is less important (reducing false positives)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### F1 score\n",
    "\n",
    "- The F1 score is a metric that \"averages\" precision and recall (not exactly the average, see [here](https://en.wikipedia.org/wiki/F1_score)).\n",
    "- If both precision and recall go up, the F1 score will go up, so in general we want this to be high.\n",
    "- Sometimes we need a single score to maximize, e.g. when doing hyperparameter tuning via `RandomizedSearchCV`.\n",
    "- Accuracy is often a bad choice.\n",
    "- Later today we'll see another good choice (ROC AUC)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a bunch of functions in sklearn to make life easier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "   Not Fraud       1.00      1.00      1.00     48333\n",
      "       Fraud       0.84      0.64      0.73        84\n",
      "\n",
      "    accuracy                           1.00     48417\n",
      "   macro avg       0.92      0.82      0.86     48417\n",
      "weighted avg       1.00      1.00      1.00     48417\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_valid_fold, pipe_lr.predict(X_valid_fold), target_names=(\"Not Fraud\", \"Fraud\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that the scores depend on which one you consider the \"positive class\". \n",
    "- If non-fraud is the case you care about, then your precision and recall are very close to 100%\n",
    "  - But not exactly 100%, see below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2XElEQVR4nO3dd5xcVf3/8dc7uymkJ5AEEggJBFAwgF9RiD8ERCUoXbpKlSIlIiJNWihKEQUVQhEFASkCiqBSjBQLCQpICxBaQggkpPe++/n9ce4mk9nZ3bthM7vZfT8fj/uYnXs/c++Z2WQ+e8o9RxGBmZlZS9KuuQtgZmZWzMnJzMxaHCcnMzNrcZyczMysxXFyMjOzFqeyuQtgH98GvSti0Cbtm7sY1ghvvty5uYtgjTSf2TMios+avn74F7vEzFlVuWKff3npYxGx55peqzVwcmoFBm3Snv88tklzF8MaYXj/7Zu7CNZIo+P+9z7O62fOquI/jw3MFVux0VsbfJxrtQZOTmZmZRBANdXNXYx1hpOTmVkZBMHyyNesZ05OZmZl45pTfk5OZmZlEARVni4uNycnM7MyqcbJKS8nJzOzMgigyskpNycnM7Mycc0pPycnM7MyCGC5+5xyc3IyMyuDINys1whOTmZm5RBQ5dyUm5OTmVkZpBkiLC8nJzOzshBVqLkLsc5wcjIzK4M0IMLJKS8nJzOzMkj3OTk55eXkZGZWJtWuOeXm5GRmVgauOTWOk5OZWRkEoop2zV2MdYaTk5lZmbhZLz8nJzOzMgjEsqho7mKsM5yczMzKIN2E62a9vJyczMzKxAMi8nNyMjMrgwhRFa455eXkZGZWJtWuOeXm5GRmVgZpQIS/cvPyJ2VmVgYeENE4Tk5mZmVS5fuccnNyMjMrA88Q0ThOTmZmZVLt0Xq5OTmZmZVBmvjVySkvJyczszIIxHJPX5Sbk5OZWRlE4JtwG8GflJlZWYjqnNsanV16VFJIuqxofy9Jt0iaIWmhpNGShpZ4fSdJP5E0RdJiSWMk7VIirp2kcyVNlLRE0kuSDqyjTMdLekPSUknjJX0n7/txcjIzK4Mg1ZzybI0l6XBguxL7BTwE7AmMAA4E2gNPStq4KPzXwPHAhcDewBTgMUnbF8VdCowErgO+CowF7pP0taJrHw/cBDyQXf8+YJSkk/K8JzfrmZmVydoYECGpJ3ANcDpwV9HhfYGdgd0j4sksfgwwATgL+G62bzvgG8CxEXFrtu9pYBxwSXYeJPUFfgBcERFXZ9d4UtIQ4Argr1lcJfAj4I6IOK8grj9wqaRbImJ5fe/LNSczszIIRHXk2xrpKmBcRNxd4ti+wIc1iQkgIuYCDwP7FcUtB+4tiFsB3AMMl9Qx2z0c6ADcWXSdO4GhkgZnz4cBfUrE3QGsT0qY9XJyMjMrgwCWR2WuLS9JOwNHAifXEbIN8GqJ/eOAgZK6FsRNiIhFJeI6AEMK4pYCb5eIA9i6II4S1y6Oq5Ob9czMykKNWc9pA0nPFTy/OSJuXu1sUntSn87VETG+jvP0BiaW2D8re+wFLMjiZtcT17vgcU5ERI44SpyzOK5OTk5mZmUQNGqGiBkRsUMDMWcD65H6duqi7NKl9q/tOOqIzcXJycysTJpqJVxJA4HzgOOAjgV9QmTPewLzSTWVUrWUXtljTc1mFjCwnrhZBY+9JKmo9lQqjuzaUwriehcdr5P7nMzMyiBCVEe7XFsOmwGdSAMOZhdskEbTzQaGkvp4tinx+q2BSRGxIHs+DhgsqXOJuGWs6mMaB3QENi8RB/BaQRwlrl0cVycnJzOzMkgDIipybTm8CHyxxAYpYX2RlFAeAgZI2rXmhZK6A/tkx2o8RLr/6eCCuErgUODxiFia7X6UlKy+WVSebwGvRsSE7PkYYEYdcbOAfzf0Bt2sZ2ZWFmqy6YsiYg7wVK0rSADvRcRT2fOHSIniTklnkmpU55L6hK4qON+Lku4Frs0GWkwATgIGU5BgImKapGuAcyXNB14gJbDdKRiaHhHLJV1Auun2A2B0FnMsMCIiljX0Hp2czMzKIA2IKO9igxFRLWlv4GpgFKkpcAzwxYh4vyj8GNLgisuAnsBLwJ4R8UJR3HmkEX6nARsC44FDIuLhomvfKCmAM4AzgUnAqRExKk/ZnZzMzMpkbS+ZEVE7+0XELFKN5dgGXrsY+H621RdXRUpgl9UXl8XeRBru3mhOTmZmZVAzQ4Tl4+RkZlYm1R6DlpuTkzWpH35jM55/qjuHnzaVo8+eunL/xPGduP2qDXn9hS4snNeOfpssY/hhszjguOlUZP8KP5rcnlHnb8y749ZjzsxKOq1XzaBPLOGQUz7is7vPX3muye905OHbNuClZ7oy5b0OdO5azZbbLeLIs6aw+TZLVivP1d/bhDde6MLMqe2proaNBi3jq4fPZO+jZ1Dhdd8abYONlnHIKdPYYtvFbLbNYjqtFxz5uU/y0eQOq8V17bGC4y6Ywuf3nEvHTsHrz3fmxov6M/GN9Zqp5M0vApZXOznl5eRkTebJP/Zkwmu1v3xmTq3krAOHsP6Gy/nOxR/Qo/cK/vevrtxyaX/mzKjkuPPTPXqLF1bQo/cKjjp7ChtstJxF89vxyF3rc/63NueCWyaw89fmAvD809146ZmufPngWQwZupiF8yq4b1RfTtt7S67501tsse3ilddetqQd+x07nY0GLUPAc09344YLB/DhxI6cdOkHZflcWpP+g5axyz5zeevl9Xj12S7ssNuCElHBxbdNpN8myxh1/gAWzKng0BHTuOr+dzj5K1syY0qHEq9p/VKznpNTXmVNTpKOBm4F5gKDI2J2wbFK0qy4F0fEyDKXaxhwLfApoDPw6Yh4sZxlKCUb6VL2z2NNLJhbwU0jB3DiyA+44pRBqx17dnQP5s6q5Gd/eouNN0+3S2y/8wKmTOzI3+/vvTI5DdpqCd//2eoDiHb88jyO3HFrHr+n98rktNt+s9n3mBmooPl++/83nyN33Jo/3tKHs34xaeX+H9743mrn+8xu85k1tT2P3dPbyWkNvDK2C4dtl+6r3PMbM0smp2HD5/GpHRdy1kGb89IzaV7R157vwu1jX+fgk6dzwwUDylrmlqSpZohoC5orjfcgzQvVUvyalKj3IU31/mbzFmfdc8tlG7HpVkv44gFzah1bviz9h+zcrWq1/V16VFFdXf95KyqhS/cqKtuvmimlx/pVqyUmgC7dqxmw2VJmTm3fYFm79VpBReUaT/nVppUYDFbLTnvMY8aUypWJCWDR/ArG/q07w4bPXZvFa9FqhpKvhSUzWqXmSk6PAyMkbdhM119JUjtgK+AvEfFERIwtMW18TWzHUvvbulef7cLo+3tz6o8nlzy+yz5z6NF7BdeftzFTJ3Vg4fx2/PuRHvz9/l4ceOL0WvHV1VC1AmZNq+R31/Tjg3c7ss/RM+otw7zZFUx8oxObDFlS61hEOt+CuRX88y89GH1fb75+Qu3rWtPYdMslTBzfqdb+997sRL+Nl9Opc1WJV7UFTTp9UavXXJ9Czfj48+qNAiR9LlvzfoGkhZL+LulzRTG3SZos6dOS/ilpkaS3GlqvPmtmrCJ9DhdICkkTi845TNIzkhaT3VEt6TBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUml5sNqcVYsh5+fvTEHfWcamwxZWjKmV58VXPPwm0x6qyNH7bQ1X99qWy49bhCHnDKNQ06ZViv+lkv787WB23P49p/ivlF9OfeG9/j0F0r1bawy6vwBEOKA42snnWdHd+drA7fnwE8O5UcnDGLfY2fwzdM/WrM3bA3q1nMFC+bW7jGYP6ciO95WkxNUo1ybNd+AiCmk9ee/J+nqiHivVJCkbYGnSZMEHk2qGZ8DPC1pp4h4qSC8O2mJ4mtJywofA9wgaXzhKpBF/kJakfFfpKa9W0gLadXoQVoJ8mrgh0BNT/tmwP2kZYmrgV2AWyStFxE35vwMCo3Mzv8zUq1yB1af96rF+v31/Vi2pB2Hf7fuL/s5Myu49LjBdOpczfm/mkD3XlW8+K+u3P3zfrTvEBx66uoJ6oDjp7Pb/rOZNa09o+/vxRWnbMr5N09kp6/MK3n+e37Zlyf/2Jvv/3QSAwbXnhVl6I4L+OUj41k4r4L//asbD9zYByk45pypJc5mH5eUaqu19pe/KC1KGq3nIaJ5NedovSuBE4GLqPvO5QtJyeJL2VxSSPobafGsi4CvF8R2A06uSUSS/gHsARwOlExOETFdUs2gjMkRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEWkuqkYlJ0m9gNNJi4n9INv9uKQqUvKr63UnACcADBzQPL/GaZPbc/cv+nH61e+zfFk7lhfkheVL27FgbgXrda3ivuv78dH7Hbj9P6+t/Kt5u88voLoabv/Jhux5+Ex6rL/qr+k+/ZfTp/9yYDE7fWUeZx44hF9d0r9kcvrz7etz6+X9OersKQw/vPQs/F26V7Pldunvik9/YQHt2wd3XduPfY6ayQYbLW+6D8QAmD+nkm49V9Ta3zX73dfUoNoa34TbOM3WuJlNqfFT4EhJW9URtgvw55rElL1uHqlWsWtR7KLCGlI2i+5blF6jJK8VwJ+Ld0raQtLd2YSGy7PtOFLfVWMNBboAvy/af099L4qImyNih4jYoc/6zfOffcqkjixb0o4rT92UAz85dOUGcP+NfTnwk0OZ+Pp6THijE/0HLa3VnLPV9otYsbwdH06svytvi20XlYwZfX8vrjt3Yw48cRrfOC1/M90W2y2iulpMndQ2hzSvbe+92ZFNt6zdxDtwiyV8NLk9Sxa1zeQEbtZrjOa+z+kaYASpGa54anWovVBVjamsWtyqRqklhpeSJjpcU9OyeaRWktQV+BuwiNTE+A5pCvmTaGDuqjpslD0Wf7u2+E6RzbdZzFX3v11r/1kHDeFLB85i+OGz6D94Kb37ruD157owf07Fagnqjf91AWD9DeuuvVRXw7j/dqH/pqt/2f37kR789PSB7PmNmZxw0YeNKvcrY7oiBRttWrqPzD6eMY/3YPhhsxm60wJeGZtG7HXuWsVOX5nHkw/2bN7CNaPmmPh1XdasySkiFki6nFSD+kmJkFmkWW+LbUiOlRSbQKnxxsOATYEvRMS/anZm92kVqhk2Vvzn+fpFz2uSbz9WLdBV87xF69qjiu0+X3qgQt+Nl608ttcRM3jiD7344eGbc9BJ0+jeawUvj+nKAzf24f99dQ59B6TkdMfVGzJ/TgVbf3YhvfsuZ9a09jx2d2/G/68z51y/qlvylbFduPzkTdnsk4v5yiGzeP35Veujte8QDBmamvCeHd2dx+/tzU5fmUufActZvKAd/32yO4/cuT5fO2Im629Yu+nJGrbzXnMA2CL7nD+7+zzmzKxk7sxKXhnblbGPdee15zpz9nWT+NWlG628CVeC+0b1bcaSNz+PxMuvuWtOkKZx/z6lZ7h9GthLUreImA8gqRvpfqSnylbC1dV8E678cz/rN9qvKO4jUs3tU0X79yp6/jKwEDgEeKJg/2Efu6QtxCc/s4if/vEtfnfNhtx44QAWzk/TF33z9I848MRVgyGGDF3EH2/pw1N/6smi+RX06rOCzbZezE//+DbbfG7hyrgX/9WN5Uvb8farnfn+fluudq1+Gy/j9v+kRTY32nQpUQ23XbkRc2dW0qV7FQMGL+XMX0xit/1LVbQtjwt+tfr4pRFXpJuZX3qmC2cdNIQIceGRgzn+wg8Z8eMPaN+pmtef68JZB2/O9A/bblNqhFjh5JRbsyeniFgq6RLg5hKHLwX2Bv4u6UpSTeZsUoK4pHylXM0zwDzgekkXkfqLziet+tijJigiIlu869uS3iStebIXsFvhySJiTrZ413nZ4l2PA58Fvl2G97JWPPbhi7X2ffIzi7jsznfrfd2w4fMYNrz0iLxCR/xgKkf8oOGRdgO3WMqFv57YYJw1zvD+2zUYM39OJT/7/kB+VobyrEvcrJdfS0njt5IGL6wmIl4mfZnPA34L3EFa5GrXomHkZRMR04EDgArScPLLSUPQ7ywRfhrwB9JQ8XtJ/V8jSsSNBH4MHEEa7LEHqXZoZq2EZ4honLLWnCLiNuC2EvtXAFsW78+OPQt8uYHzHl3H/t1ylGkFJW7BqOuc2bEngE+XODSyKG4OKeEUU1FcFan2dX59cWa2bnPiya/Zm/XMzNoC3+fUOE5OZmZl4nuY8nNyMjMrgwhY4cUGc3NyMjMrEzfr5efkZGZWBu5zahwnJzOzMsmzWKMlTk5mZmXiARH5OTmZmZVBhPucGsPJycysLESVR+vl5uRkZlYm7nPKr87kJKma0ktGlBIR4URnZlYHr+fUOPUllEvIn5zMzKw+kfqdLJ86k1NEjCxjOczMWj2P1suv0U1x2TLl6wMfRkTd62ubmdlK4QERjZL7k5K0t6QXgLnAu8DQbP8tkr6xlspnZtZqROTbLGdykrQ/8CfSaq9ns/o6QxOAo5q8ZGZmrUyEcm0NkTRc0hOSpkpaKmmypN9L2roorldWgZghaaGk0ZKGljhfJ0k/kTRF0mJJYyTtUiKunaRzJU2UtETSS5IOrKOMx0t6IyvfeEnfacRHlbvmdBFwa0TsAVxbdOxV4FONuaiZWVuTakVNk5yA3sDzwKmklbPPBbYBxkraFECSSCtr70lagftAoD3wpKSNi873a+B44EJgb2AK8Jik7YviLiUtqnod8FVgLHCfpK8VBkk6HrgJeCC7/n3AKEkn5XlzkL/P6ZPAWdnPxZXO2aQ+KDMzq0dTDSWPiLuBuwv3SfoP8AZwEPBTYF9gZ2D3iHgyixlDau06C/hutm874BvAsRFxa7bvaWAcadT2vtm+vsAPgCsi4urssk9KGgJcAfw1i6sEfgTcERHnFcT1By6VdEue8Qp5a07zgA3qODYImJ7zPGZmbdZa7nOamT3WfPHvSxq49uSq68dc4GFgv4LX7Zu95t6CuBXAPcBwSR2z3cOBDsCdRde9ExgqaXD2fBjQp0TcHaSKzM553kze5PQ34FxJPQv2RVboU4FHcp7HzKxNCkR1dbtcW16SKiR1kLQFqRltKimpQGrme7XEy8YBA7OR1zVxEyJiUYm4DsCQgrilwNsl4gC2LoijxLWL4+qVt1nvPOA/wHhS1S2Ac4BtgR7A/jnPY2bWZjWiUrSBpOcKnt8cETeXiHsW+Ez289ukJrxp2fPewMQSr5mVPfYCFmRxs+uJ613wOCeiVt2uVBwlzlkcV69cySkiJkr6P+BiUtWuCtgFeBS4MCI+zHMeM7M2Kxo1t96MiNghR9wRQHdgM1J/0N8k7RwRE0mjqkvlw+JCrI046ojNLfdNuBExGfj2x7mYmVmb1sT3MEXE69mPz0p6hFRTOgf4DqmmUqqW0it7rKnZzAIG1hM3q+CxlyQV1Z5KxZFde0pBXO+i4/Vq9O3KkvpL+mw28sLMzHJqwqHkJc4dc0hNezV9RONY1f9TaGtgUkQsKIgbLKlzibhlrOpjGgd0BDYvEQfwWkEcJa5dHFevxswQcaSkCcD7pLHt70uaIOlbec9hZtZWBVBdrVzbmpDUD/gE8E626yFggKRdC2K6A/tkxyiIaw8cXBBXCRwKPB4RS7Pdj5KS1TeLLv0t4NWImJA9H0OasKFU3Czg33neT65mPUmnAr8ARpNuwvoI6AccDvxWUo+IuD7PuczM2qQAmug+J0l/BF4AXibd6rMlcDqwgnSPE6SkMwa4U9KZpGa8c0l9QletLFbEi5LuBa6V1J50H9RJwGAKEkxETJN0DWnk9vzs+ocCu1MwND0ilku6gHTT7QekvLE7cCwwIiKW5XmPefuczgBui4hji/b/RtJtpI44Jyczs3o04bx5Y4FDSN/NHUgtWk8Bl2eDIYiIakl7A1cDo4BOpGT1xYh4v+h8x5BunL0M6Am8BOwZES8UxZ1HGuF3GrAhaQT3IRHxcGFQRNwoKbLynQlMAk6NiFF532De5LQhq8bOF7uL9CGZmVl9mig5RcSVwJU54maRaizFFYviuMXA97OtvrgqUgK7LMe1byLde7VG8ianV6jdCVZjC0rf6GVmZiut+WCHtihvcjoNuEfSDOAPEVElqYI0keCZwGFrq4BmZq2Gl8PIrc7kJOl9Vv8oe5Ca9qokzSaNba8gtT/eC2y6FstpZrZuC4g1HInXFtVXc/o7zvNmZk3IySmvOpNTRBxdxnKYmbV+/nM/t9zTF5mZ2cfk5JRbo5JTtijVVqTx8quJiNubqlBmZq1OE96E2xbknSGiJ/AXYKeaXdlj4d8BTk5mZvVowptwW728c+v9mLSC4S6kxHQAaTqK3wHvAp9bK6UzM2tNqpVvs9zJaTgpQY3Nnk+OiKci4kjSvEmnrY3CmZm1Jop8m+VPThsB72ZTVywBuhUc+wOwV1MXzMysVYlGbJY7OU0lTQYI8B4wrODYkFrRZmZWRGlARJ7Nco/W+xcpIf0ZuAO4SNIg0vTsR7H62iBmZlaKa0W55U1OFwM1K9/+hDQ44lCgMykxjWj6opmZtTLVzV2AdUeu5BQR75CtrhgRy0lrdJyxFstlZta6+D6nRvEMEWZmZeKRePnVNyv5hY04T0TEpU1QHjOz1svJKbf6ak4jG3GeAJyczMysSdQ3K3neYebWzN58uTPD+2/f3MUwswa4WS8/9zmZmZVD4KmJGsHJycysXFxzys3JycysTNysl5+Tk5lZuTg55ebkZGZWLk5OuTk5mZmVgZfDaJzGLtO+LWnBwfWBmyJiqqQhwEcRMX9tFNDMrNXwaL3c8i7T3hG4E/g6aSXcAB4mLaVxFfAmcM5aKqOZWavgmlN+eW+0/RHwZeAIoB8pQdV4hLRSrpmZ1ceLDeaWt1nvcOD8iLhLUkXRsQnAoCYtlZlZa+M+p0bJm5zWB16v41g7oGPTFMfMrBVzcsotb7PeBFZfmr3Q54DxTVMcM7PWS9X5NsufnG4HzpH0TaBDti8kfRE4HfjN2iicmZm1TXmT01XAX4A7gFnZvn8Bo4FHI+KXa6FsZmatSxMNiJB0kKQHJL0nabGk8ZIul9StKK6XpFskzZC0UNJoSUNLnK+TpJ9ImpKdb4ykXUrEtZN0rqSJkpZIeknSgXWU8XhJb0hampXvOw2/s1VyJaeIqIqIw4BdgZ8CtwC/AHaPiG825oJmZm1SrLoRt6Ethx8AVcAPgT2BG4CTgL9JagcgScBD2fERwIFAe+BJSRsXne/XwPHAhcDewBTgMUnbF8VdSlrr7zrgq8BY4D5JXysMknQ8cBPwQHb9+4BRkk7K9e5o5E24EfFP4J+NeY2ZmWWabkDEPhExveD505JmAb8FdgOeAPYFdiZVIp4EkDSGNIbgLOC72b7tgG8Ax0bErdm+p4FxwCXZeZDUl5QUr4iIq7PrPplNxHAF8NcsrpJ0+9EdEXFeQVx/4FJJt0TE8obeoBcUNDMrlyZq1itKTDX+mz0OyB73BT6sSUzZ6+aSJlDYr+B1+wLLgXsL4lYA9wDDs0kYIN3P2oE0IUOhO4GhkgZnz4cBfUrE3UEa+b1zQ+8PciYnSdWSqurb8pzHzKytEmt9tN6u2WPNbT/bAK+WiBsHDJTUtSBuQkQsKhHXARhSELcUeLtEHMDWBXGUuHZxXL3yNutdQu18vj6wB+kep9tynsfMrG1q3E24G0h6ruD5zRFxc13BkgaQvqdHR0TN63oDE0uE1wxq6wUsyOJm1xPXu+BxTkQUv4tScZQ4Z3FcvXIlp4gYWWp/NlvEw8DcPOcxM2vT8ienGRGxQ57ArAb0J2AFcEzhoTquWDz77NqIo47Y3D5Wn1NEVAGjgO99nPOYmbUJTTy3nqROpBF5mwHDI2JyweFZlK6l9MoeZ+eMm1Xw2CsbBdhQHCXO2bvoeL2aYkBExxKFMDOzIk04lBxJ7UlDtT8HfC0iXikKGceq/p9CWwOTImJBQdxgSZ1LxC1jVR/TONL3/eYl4gBeK4ijxLWL4+qVd0DEwBLbEEn7k4YQPtfAKczMrOluwm0H/A74ErBfRIwtEfYQMEDSrgWv6w7skx0rjGsPHFwQVwkcCjweEUuz3Y+SklXxva3fAl6NiAnZ8zHAjDriZgH/bvgd5h8QMZG62xrfAU7JeR4zs7YpmnTevOtJyeRHwEJJOxUcm5w17z1EShR3SjqT1Ix3Lul7+6qVxYp4UdK9wLVZbWwC6YbewRQkmIiYJuka4FxJ84EXSAlsdwqGpkfEckkXkG66/YA0k9DuwLHAiIhYlucN5k1Ox5TYtwR4D/hv1vdkZmb1abqbcL+aPZ6XbYUuBkZGRLWkvYGrSWMDOpGS1Rcj4v2i1xxDSnSXAT2Bl4A9I+KForjzSCP8TgM2JE36fUhEPFwYFBE3SgrgDOBMYBJwakSMyvsGG0xO2Yi8F0k3c5W68cvMzHJoqvWcImJQzrhZpBrLsQ3ELQa+n231xVWREthlOa59E2kKozWSp88pSH1Kn17Ti5iZGV4JtxEarDllVcP3gS5lKI+ZWevkxNMoeYeS3wR8T1KHBiPNzKwW0bRDyVu7vAMiupHGtr8r6VHSdOqFH2FExEVNXTgzs9bEiSe/OpOTpHeBAyLiJdKaITVKdawF4ORkZlYfJ6fc6qs5DSLdDUxEeGkNM7OPy8kpt0YtNmhmZmvI/UmN0lBy8kdpZtZU/I2aW0PJ6WJJM3KcJyLiqKYokJlZa9WE0xe1eg0lp+1JKx82xH8PmJk1wM16+TWUnPaPiP+UpSRmZq2Zb8JtFA+IMDMrFyen3JyczMzKoGaGCMvHycnMrExU7eyUV53JyTfempk1Ifc5NYprTmZmZeJmvfycnMzMysXJKTcnJzOzMnHNKT8nJzOzcnFyys3JycysHMLTFzWGk5OZWRn4PqfGcXKyZvOZXedxyCnTGbjlErr2qGLuzEpef64zd/x0Qya91WllXNceKzjugil8fs+5dOwUvP58Z268qD8T31ivGUvf9mw7bAE/eeCdWvsXzG3HgZ8cWvI1373yffY6YhZ/f6AnV43YdG0XseULZ6e82nRyknQ0cGsdh78SEaPLWJxaJD0FEBG7NWc51pZuvap465X1ePi36zN3ZiV9Byzj0FOnce2f3+I7u2/FtA86AMHFt02k3ybLGHX+ABbMqeDQEdO46v53OPkrWzJjSofmfhttzvXn9+fNFzuvfF61QiXjtt5hIbt/fQ4L5/mWyRquOeXXppNTgYOByUX7XmuOgrQlTz3Yi6ce7LXavvEvdubX/xzPF/aewwM39WXY8Hl8aseFnHXQ5rz0TFcAXnu+C7ePfZ2DT57ODRcMaI6it2nvv9WJN17oUm9MRWVw2lWTufvnffnaETPLVLIWzjfhNor/pElejIixRdu84iBJHZujcG3JvNnp76UV2V/jO+0xjxlTKlcmJoBF8ysY+7fuDBs+t1nKaA07+KRptKsIHripT3MXpUVRdb7NnJzqJOloSSFpF0n3SZoDPJsd+6yk+yVNlrRY0nhJP5a0XtE5Jkq6rcS5Q9LIon2HSXpD0lJJ4yQdsPbeXcvSrl1Q2b6a/oOXctqVk5n5USVPPdgTgE23XMLE8Z1qvea9NzvRb+PldOpcVebS2tnXTeKv77/Efa++yjnXv0efActWO77Rpks5/Hsfcd25G7Niub9iCjk55edmvaRCUuFnUVj5/h1wN3AQqz6vgcCLwG3AfGAb4EJgM+Cwxl5c0peBu4C/AGcAfYCfA+2B8Y0937rm5395iy23WwzAB+924OyDN2fuzPYAdOu5go8md671mvlzKrLjVSxZVFG+wrZhC+e34/4b+/DymC4sml/BkE8t5rDvTuPah97i5D22XPk7++6Vk/n3X3usVts1smY9t+vl5eSUvFH0/N/ALdnP90fEWYUHI+KBmp8lKYufB9wu6ZSIaGwj+8VZGfaLiOrsvK8DY6kjOUk6ATgBoBO1v7zXJVeNGEjnblVstOkyDvrOdC6/513O2H8IH03ugFT6/3PpLnhbm955tTPvvLrq39orY7vyyrNd+MVf3mL/b8/gt1dtxO5fn82W2y3iuF0+0Ywlbbk8ICI/17mTA4DPFmzfLjj2x+JgSd0lXSnpHdIy9suBO0jfmVs05sKSKrJr3l+TmAAi4llgYl2vi4ibI2KHiNihPet2V9j7b3di/P+68NSDvTj7kM1Zr0sVh546DYD5cyrp1nNFrdd07VmVHXetqTm9/UpnJr/bkS23X0SnzlWcOPJDfn99X5YtbUeX7lV06V5Fu3ZQ2T7o0r2Kiso2/u0cOTdzzSnzakS8XbhD0rDsxykl4m8FvkxqynsRWAh8DrgeqN1BUr8NSM13H5U4Vmpfq7ZwXgUfTuxI/8FLAXjvzY783y4LasUN3GIJH01u7ya9FkACQvToXUXPDVZw7A+ncuwPp64W03fAXHbddy4jjx3EmEd7NE9Bm5lvwm0cJ6eGrfbPSVInYD9gZET8vGB/qbsQlwCr3YgjqXdRzAxSzatfidf3A95bgzKvs3pusJxNhizliT/0BGDM4z0Yfthshu60gFfGpj6Mzl2r2Okr83gyGzRhzWeLbRcxYLOl/OPhnsyaXsmZB25eK+bcG95j4huduPvn/UoObmkzIrzYYCM4OTVeR6CClFAKHV0i9j3gU0X79i58EhFVkv4LHCRpZEGf047AIFpxcrrw1xN4+5XOTHi9E4vmVzBgs6V8/YTpVK2AB27qC8DYx7rz2nOdOfu6Sfzq0o1W3oQrwX2j+jbzO2hbzr7uPaZO6sDbr6zHgnlpQMShI6Yxc2p7/vSbDVi+tB0vj6k9CGLZUjF7emXJY22Oc1NuTk6NFBFzJY0FzpA0hVTzORYodTfoPcBvJF0D/BnYjtJJ7CLgceBBSTeRRutdDEwtEdtqvPFCF3bZZw4HnjiNyg7B9A878PIzXbj3l/34aHKqcEaIC48czPEXfsiIH39A+07VvP5cF846eHOmf+jZIcpp4hud2G3/Oex37Aw6rlfN7Ont+fdfe3DH1Rsyb5a/SvJoymY9SRsDZwM7kL5b1gMGR8TEorhewE+A/bOYMcDpEfFKUVwn4FLgW0BPUpfF2RHxj6K4dtl1TwQ2JA3auqRwoFhB7PGkEciDSX3o10TEjXnen/9FrZnDgRtIfUyLgd8Dp5ESUKHfApuQBlicCPyTNPhitf6tiBgt6ZvASOAP2fHvZedstX5/fV9+f33DtZ/5cyr52fcH8rMylMnqdu91/bj3ulKtz/U7aset10Jp1kEBNG2z3hDgEOB50nfLHsUB2Wjih0jJYQQwGzgXeFLS9hFRODPOr4G9gDOBd4FTgMckDYuIFwviLgV+AJyXXfsw4D5Je0fEXwuufTxwE3A5MBr4EjBKkiLihobenMLj7td53dU7dtSXmrsYZq3a6Lj/+YjYYU1f363HxvF//++7uWL/8cjZDV5LUruCboDjgF9RVHOStB/wILB7RDyZ7esBTADujIjvZvu2I9WUjo2IW7N9lcA4YHxE7Jvt6wu8D1wRERcVXOfvQJ+I2LbgtR8Cj0TEUQVxvwH2BTaKiOKukdV4KLmZWZko8m15FN56Uo99gQ9rElP2urnAw6SBXYVxy4F7C+JWkLomhhdM3TacNMjrzqLr3AkMlTQ4ez6M1D1RHHcHsD6wc0MFd3IyMysTVUeurQltA7xaYv84YKCkrgVxEyJiUYm4DqQmxJq4pRR1TWRxAFsXxFHi2sVxdXJyMjMrh7w34KbctIGk5wq2E9bwqr1J/UzFZmWPvXLG9S54nBO1+4NKxVHinMVxdfKACDOzMkg34eauFc34OP1bRZctddHiGcDWRhx1xObimpOZWblU59yazixK11Jqakyzc8bNKnjslY0CbCiOEufsXXS8Tk5OZmZloohcWxMax6r+n0JbA5MiYkFB3GBJxbNIbw0sY1Uf0zjSRATFU4HU9CG9VhBHiWsXx9XJycnMrBwa1+fUVB4CBkjatWaHpO7APtmxwrj2pFXBa+IqgUOBxyNiabb7UVKy+mbRdb5FmqN0QvZ8DGmCglJxs0grOdTLfU5mZmXR9HPrSToo+/Ez2eNXJU0HpkfE06SkMwa4U9KZrLoJV8BVK0sW8aKke4FrJbUn3Qd1Eunm3W8WxE3LZrw5V9J84AVSAtudgqHpEbFc0gWkm24/IN2EuztpNp0REbH6CpUlODmZmZVL0096cF/R81HZ49PAbhFRLWlv4OrsWCdSsvpiRLxf9NpjgB8Bl5GmL3oJ2DMiXiiKOw9YQJrBpmb6okMi4uHCoIi4UVKQpi86E5gEnBoRo8jBycnMrByi6Zdgj4gG192MiFmkGsuxDcQtBr6fbfXFVZES2GU5rn0TaQqjRnNyMjMrF08Xl5uTk5lZuTg35ebkZGZWJqpu4na9VszJycysHIKmvsG2VXNyMjMrA9HkN9i2ak5OZmbl4uSUm5OTmVm5ODnl5uRkZlYO7nNqFCcnM7My8Wi9/JyczMzKItys1whOTmZm5RA4OTWCk5OZWbm4VS83JyczszLxfU75OTmZmZWLk1NuTk5mZuUQAVVu18vLycnMrFxcc8rNycnMrFycnHJzcjIzK4cAqp2c8nJyMjMri4Bwn1NeTk5mZuUQeEBEIzg5mZmVi/uccnNyMjMrFyen3JyczMzKwhO/NoaTk5lZOQTgJTNyc3IyMysX15xyc3IyMysLT1/UGE5OZmblEBC+zyk3Jyczs3LxDBG5OTmZmZWL+5xyc3IyMyuHCI/WawQnJzOzcnHNKTcnJzOzsgiiqqq5C7HOcHIyMysHL5nRKE5OZmbl4qHkuTk5mZmVQQDhmlNuTk5mZuUQXmywMZyczMzKxAMi8lN4aOM6T9J04L3mLsdasgEwo7kLYbm15t/XphHRZ01fLOlR0ueTx4yI2HNNr9UaODlZiybpuYjYobnLYfn492VNpV1zF8DMzKyYk5OZmbU4Tk7W0t3c3AWwRvHvy5qE+5zMzKzFcc3JzMxaHCcnMzNrcZycDElHSwpJcyT1KjpWmR0b2QzlGibpWUkLszJsX+4ylNJcn0dTK/i9l9q+3ALK95Skp5q7HNY8PEOEFeoBnA2c09wFyfwaWAzsAywC3mze4rRaBwOTi/a91hwFMavh5GSFHgdGSLo2IqY2Z0EktQO2An4UEU80ENsxIpaWp2St0osR8XZDQf6crZzcrGeFLssez2soUNLnJI2WtCBrdvu7pM8VxdwmabKkT0v6p6RFkt6S9J0Gzn00UEX693lB1sw0seicwyQ9I2kxcFV27DBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUnbNPTZtAYFTX67SLpP0hzg2ezYZyXdn/0eFksaL+nHktYrOsdESbeVOHetZtHsd/eGpKWSxkk6YO29O1sXODlZoSnAdcAJkjatK0jStsDTQC/gaOBIoDvwtKTtisK7A3cBdwL7Af8FbpD0xXrK8Rdg5+znXwPDgMIvqx7APcDdwFez8wNsBtwPfBPYH3gYuKWhZFiPkcAPgd9l53sceGgNz9WSVWR9izVbRcGx3wETgINY1dw7EHgR+A6wJ/Bz4Fjg1jW5eNa/dRfwFvB14CfZObdak/NZ6+BmPSt2JXAicBHpC6eUC4GlwJciYg6ApL8BE7PXfb0gthtwckQ8mcX9A9gDOBx4stTJI2K6pNnZ08kRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEXAScGNdb7iUbGDI6cDNEfGDbPfjkqqAKxpzrnXAG0XP/w3ckv18f0ScVXgwIh6o+VmSsvh5wO2STomImY28/sVZGfaLSGtKSHodGAuMb+S5rJVwzclWExGzgJ8CR0qq6y/XXYA/1ySm7HXzSLWKXYtiF9UkpixuKekv5IEfo5grgD8X75S0haS7JX0ALM+241izv8CHAl2A3xftv2cNztXSHQB8tmD7dsGxPxYHS+ou6UpJ75D+SFkO3AEI2KIxF85qaZ8lJcGVix1FxLOkP3asjXJyslKuAWYBl9RxvDepCbDYVFJTX6HZJeKWAp3WuHQwLSJWWxhHUlfgb8B2pOanL5C+9H4DdFyDa2yUPX5UtL/4eWvwakQ8V7AV1lZK/Z5vJTXp/QL4CulzPiU71tjf6wZAe0p/rq3xs7ac3KxntUTEAkmXk2pQPykRMgvYsMT+DbNja1upObeGAZsCX4iIf9XslFT8b3xJ9tihaP/6Rc9rvpT7AeMK9vdrXFHXeat91pI6kfoOR0bEzwv2Dy3x2iUUfc6SehfFzCDVvEp9rv1oveuUWQNcc7K6jAI+YNUIvkJPA3tJ6lazI/t5n+xYc+icPS6v2ZH1G+1XFPcRqeb2qaL9exU9fxlYCBxStP+wj1fMdV5HoIKCzzlzdInY96j9Oe9d+CSrAf8XOCjrJwRA0o7AoI9ZVluHueZkJUXEUkmXUHqW6UtJXzJ/l3Ql6a/rs0kJoq6mwLXtGVKn/PWSLiL1F51P+su8R01QRISke4FvS3qT1OG+F7Bb4ckiYo6ka4DzJM0njdQr7o9pcyJirqSxwBmSppA+32OBASXC7wF+k32OfyY1uR5dIu4i0uf7oKSbgD6kQRLNeq+dNS/XnKw+t5IGL6wmIl4mfZnPA35L6gxfAOwaES+Vs4AFZZpO6tivIA0nv5w04uzOEuGnAX8gDRW/l9RPMqJE3Ejgx8ARpMEee5Bqh23d4cDzwPXAbaQkclqJuN+yavTmw8BwVr8lAICIGE0a/r8V6fdyJvA9PFKvTfOSGWZm1uK45mRmZi2Ok5OZmbU4Tk5mZtbiODmZmVmL4+RkZmYtjpOTmZm1OE5O1mKUWDZ8vqSXJJ1aYhqipr52rXWesrWjJjbyPLtJGlk420ETlW+kpAbv+6hrDaW852+qz7ngdzmoKc5nbY+Tk7VEB5PmyjsQ+A/wS9IyHeV2KSVuGm3AbqQbT/1/y+xj8PRF1hIVLhv+uKQhpBkDSiYoSe2BFdHEd5RHxDtNeT4zy89/3dm64L9AN0l9C5rfTpZ0laQPSRO59gSQ9HVJY7Nl1edkS4yvtnaUpM6SRkmaqbSc+0PAxsUXLdWsJ6mLpCskvZMtKT5V0gOS+mVLj1+UhS6vaZ4suu6VkiZIWpY9nlfcBKhVy9ovkfSBpAtIayU1mqQ+km6S9Gb2mbwv6S5JpebCA/ikpCez2CmSLilRvg0k3ZCVbanS8uonrEn5zOrimpOtCwYDVaT5+2pmHz+PlLROIM2nt0RpOfYbSHMCXkJahXckafn4bSNifvbam4BDSZOL/pe0JlHNUu91ktSBtGbU9qS5+8aSJpUdTlrH6hZSkvs2aZn5qoLXVgKPAVuTmgtfAXYCLiCtj3VGFrcB8ARpvrqjSIn3TNZ8ccbepKUrzgWmA/2za/1b0iciYklR/IOkNbAuz97XBUA16XNEUnfSyrfrZfsmZHE3SOoYEb9cw3KarS4ivHlrERtpxuogTQBaSfrCP5H0Jf9gFjMoi3mBbG7IbH9XYC7wm6JzDgKWAd/Lnm+Vne+corgbsvMeXbDvNmBiwfNjs5h963kPI7OYyqL9R2T7dynaf15Wvr7Z8x9lzwcWxHQhzf4dOT7DicBt9RyvADbJynJAiXIXfy6/AuYDPbPnF5CS3RYl4mbUvO+C3+Wg5v535W3d3NysZy3RG6T1gmaR1pX6HSkxFHowIgr7mIYB3YHfSaqs2YDJ2fl2yeJ2JDVnr8ny63sAUyPioca8mcyepPWNnikq3+OklWB3KngfYyNiUs0LI2IhaVbvNSLppGzU4wLSEvc15y61fH2pz6Urq9Zl2hN4FphQ9D4eIy3YuPWaltOskJv1rCU6gJRU5gPvRe2mJ6i9fHjf7HF0HeesWS7+4yy/vj5pAcY10Ze0Um/xIn2F54ZUvldLHF+jJcsljSAtp/4zUvPgbFJyHkvpJdXr+lxq+qj6AkNo+H2YfSxOTtYSvRqrRuvVpXhk3szs8WhWX1a9Rk1/U+Hy6+8WHM+z/PoMaq/smtdMUv9M8cq6NSZmj1PqKMuaLg9/GPD3iDijZoekwfXE1/W51CTlmcA0Sq/fBF6DyZqIk5O1Fs+QEtCQiPhtPXHPkjr4DwGuKNifZ/n1x4HDJO0TEXU1sy3NHtdjVUIEeJR039aCiHijnmuMAc6UtElEvA9phCBrvshhZ9KikIWOqSe+1OeygFW1uUdJCzNOiohpa1gmswY5OVmrEBHzJJ1JWqa9D/AIaYDEAGBX4KmIuCsixku6C6gZIl0zWu9rOS5zJ3A8cLeky0mJrhtptNq1WdJ5LYs9Q9IjQFVEPEfqNzuGtLT9T4GXgA7A5sC+wP4RsQi4BjiZdH/XSFaN1lu8hh/No8DZkn5IuqF5d+CgeuKPL/hchgPHASMjYk52/BrSSMd/ZsuvjycN2PgE8IWI2G8Ny2m2GicnazUi4iZJ75O+zL9BGmjwAfAP4MWC0BNJtYEfkBLEE1n8vxo4/3JJe5DuZTohe5xJGlo9Kwv7M2kQx8mkm4ZFGlW4XNJw4JzstYOBhcA7wF9II/SIiBmSvgT8nLTM+UzgRtL/1TWZJeMS0j1gp5P6mJ4mJZ1364jfjzQjxwWk5H4Zaeh7zWcwV9Lns7KcTUr+c0hJ6oE1KJ9ZSV6m3czMWhwPJTczsxbHycnMzFocJyczM2txnJzMzKzFcXIyM7MWx8nJzMxaHCcnMzNrcZyczMysxfn/aiddLxTt1IUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm_plot.plot();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "   Not Fraud     0.9994    0.9998    0.9996     48333\n",
      "       Fraud     0.8438    0.6429    0.7297        84\n",
      "\n",
      "    accuracy                         0.9992     48417\n",
      "   macro avg     0.9216    0.8213    0.8647     48417\n",
      "weighted avg     0.9991    0.9992    0.9991     48417\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_valid_fold, pipe_lr.predict(X_valid_fold), target_names=(\"Not Fraud\", \"Fraud\"), digits=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluation metrics cheat sheet (created by Varada Kolhatkar):\n",
    "\n",
    "<img src='img/evaluation-metrics.png' width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision-recall curves (10 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What if you care about certain types of mistakes more than others?\n",
    "- In the fraud detection example, you want to be conservative and bias the model towards predicting \"positive\".\n",
    "- We said earlier that `predict` returns 1 when `predict_proba`'s probabilities is above 0.5 for the positive class.\n",
    "- Key idea: what if we threshold the probability at a different values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84375"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_valid_fold, pipe_lr.predict(X_valid_fold))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6428571428571429"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_valid_fold, pipe_lr.predict(X_valid_fold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could get these predictions by thresholding the probabilities instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6428571428571429"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = pipe_lr.predict_proba(X_valid_fold)[:,1] > 0.5\n",
    "recall_score(y_valid_fold, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But in that case... what if we change the threshold?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5595238095238095"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1] > 0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Increasing the threshold means a higher bar for predicting fraud\n",
    "- So our recall goes down (the precision would normally go up, but not always - see note below)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise we could use a low threshold to get high recall (but while sacrificing precision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7619047619047619"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1] > 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7710843373493976"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1] > 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A precision-recall curve does this computation for a grid of possible thresholds and shows the results in a plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import plot_precision_recall_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEQCAYAAACeDyIUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0f0lEQVR4nO3dd3zV1f3H8dcnO2GDAVlhK0tBjFsQrQjWClortm6xDlqr/hzVOlBBratqrXXgKKDWgQsqKhQU6kAFFVBwICaylBkII4SM8/vje2/MuMkduTc34/18PPK4yfl+z72fk3E/Od9zvueYcw4REZFwJcQ7ABERaZiUQEREJCJKICIiEhElEBERiYgSiIiIRCQp3gHUpX322cd179493mGIiDQYn3766WbnXGagY00qgXTv3p3FixfHOwwRkQbDzH6o7pguYYmISESUQEREJCJKICIiEpE6TyBm1sXM/mFmC81st5k5M+seYt00M7vXzH40swLfcwyLccgiIhJAPHogvYGxQB7wXph1nwIuAiYAvwJ+BGab2eBoBigiIsHFYxbW/5xzHQDM7PfACaFUMrNBwJnAOOfcv3xlC4DlwERgdGzCFRGRQOq8B+KcK42w6migCHix3HMVAy8AI80sNQrhiYhIiBrSfSADgBzn3O5K5cuBFLxLY8tj8cIPzVtJcUmkea/xGLpfJod0bxvvMESknmhICaQt3rhJZVvLHa/CzC4GLgbIysqK6IUfW7CKgqKSiOo2Fs7BRzlbeemSI+IdiojUEw0pgRgQaPcrq6mSc24yMBkgOzs7ot2zVkwcFUm1RuWsJz+isEi9MBH5WUO6D2QrgXsZbcodFxGROtKQEshyoIeZZVQq7w/sBb6r+5BERJquhpRAZgLJwOn+AjNLAs4A5jjnCuMVmIhIUxSXMRAz+43v04N9jyea2SZgk3NugZl1A1YBE51zEwGcc0vM7EXgQTNLBnKA8UAP4Ky6bYGIiMRrEH16pa8f8T0uAIbjDYwnUrWHdAFwB3A70BpYCoxyzn0Wq0BFRCSwuCQQ51ywmVO5BJhd5ZwrAK7yfYiISBw1pDEQERGpR5RAREQkIkogIiISESUQERGJiBKIiIhERAlEREQiogQiIiIRUQIREZGIKIGIiEhEGtJ+IFKPvPv1Rjbt/Hn9yq5tMjiiV7s4RiQidU0JREKyODePwuJS7pv9DZt3FvLCojUVjicnGt9MOpGEhBpXqRGRRkQJREJSWOztRvjoglWUlHobO/7t9EEc1rMtUz7I5cn3cwJuFykijZcSiIQkMcEoKXWsuvOXVY61TE+OQ0QiEm9KIBKSQIlDRJo2zcISEZGIKIGIiEhElEBEJHRbtsCoUd6jNHlKICISuilTYPZsmDo13pFIPaAEIiKhcQ4eeMD7/IEHvK+lSVMCEZHQvPcebN/ufb5tG7z/flzDkfhTAhGR0Dz4IOza5X2+a9fPvRFpsnQfiMTUmq27OeWfH7CzsLisbNh+mTxxbnYco2qatu8uoqjUW1Fg7ooNLFu3vexYalICfzquD22bpXgFY8bAzJkVnyAl5efLVs7BrFlgFZeu2XzcSJq/PYu05MSYtUPqDyUQibpJb6zgvZWbANi9t4Qtu/byqwM70rlNOvO/3sSK9fkhPc/qLbv5fE1e2depSQkM37+93pxqUFRSWvb51A9zWbetAIAF327i+027qpy/T/NUSkpLydtdRHa3tpx0YEfvwJ13wpIlsHEj7Nnjle3dW7Fyua8LklLYktGaC7qP5txP13LO4d2i2i6pn5RApNa+37QTgL+8uozEBGPmkvW0Sk9mUNfWABzWox0TTu5Pq/Rktuzcy8JVgaeA7ikqYf43m9jrexO8/PnPq5zz0O8OYvSgTrFpSAOzp6iE6YvXsHtvCQBPvp/Dph2FVc5rkZZEcYnXc7jppH6kJnlXro/svQ+9MpuzcsMORjzwP2YuXUeJc8z7agOLc/NocfGj/Pmlezn8q4VkFFV9Xr+9qensOeFE1t/5ACuf/YKCvcXVniuNixKI1NqbX/wEwLyvNpKYYDRLTWL8sb1D+i901aadrPf9l7woZysPvfNdheN9923BP88awk/b93DWkx9TWFQS/QbUYwV7S/hsdR6lvktHd8z6ivyCIsysrHdR2TUn7AdAYkICpx3cmfYt0mp8jTV5uwGYvXwDs5dvKCsfM7gTM/5yPxvmTGfsc/eTuDdAEklNJeXB+0m59FIGFBYDX0TQSmmolECk1nrs04xvNuzgkxuPJzHIcu6lzrFuWwFrtnpvWif/4/2y/6DBu6T+yvgjaZnmLdDYpU06acmJpCRWnO+xs7CYD7/bXPbGCnBoj3Y/X8NvoIpLSrl91lfk7fYuD81Ysj7geacN6QJAWnICV5+wP2nJ3vcnLSkx7CX1j+y1DwCTzzmYnpnNAchskUor/yKZPYvglX9CNQmEgw8O6/Wk8VACkVp77Y9Hsnrr7qDJA+DVz9YBMPSed8vKzjosi1MP6gxAm2Yp9PK9iQUye/kGWqQl89iCVSxZs63CsbMPz+L2Uw4IGsPjC1ZVGA/IbJHK1Sfsh1nw+HcVFpclLTOjeWr4f0KFxSV88N1m9hZ7z3Pnm1+xeutuzCreWtG9XQZZbTNIT07kjlMH+l4TBnRqFdVxoLTkRHLvOqn6ExYvhqIiygJIT4eCAi/YoiLv+CGHRC0eaTjqPIGYWVfgAWAEYMBc4Ern3OoQ6mYBk4BjgX2AtcBLwF+dc1VHCKVOZKQk0XfflmHVufc3BwLeMvG/6NuBVhk1Lwnv3/1w7lcbmPvVz5dZ3rpiKADnPv0Je4pKA9Z95+sNTP3wB8DrAb23cjMA+7ZMY/feYvL3FHPW4Vl0bJVepe43P+1g6y6vN/DCotVVegRXj9iPP/2iT9D2PjL/O3I3e7+ib33xEzsKq44T/OnY3gCkJidyzhHdynphcffee17CSEuDDh286bxXXOENsBcUeMfHj493lBIHdZpAzCwDeAcoBM4DHHA78K6ZHVhTEjCzZnjJJhm4GVgNHALcBvQBzoht9BINd556AB+u2szp2V3Dque/nDJsv0yuH9UX8HoOmS1SASpc4tq8s5D/e3EJOwuLMeCz1dsAvEF95xiS1ZrrT+zHoT3a8uKi1Vz3yhdl//l/89MOFuVuBWDTjkL+Pm9llVhuOqkfAH+fu5K1eT+PQ2zdtZcC3xjNcx/9wKufrSMxwSgqKWWjb3C7Y6s00lMSaZaaxGPnHFwWd499mpGeUk9nl338MSQmelN7n3oKmjWDESNg3Dh45RXvuDRJdd0DuQjoCezvnPsOwMyWASuBS4D7a6h7FF6iGOmcm+Mre9fM2gLXmFmGc2537EKXaDjzsCzOPCwr7Hq9Mpvz0iVHMCSrNUmJVe9/XbetgJc/Xct3G3eWXdpqlpLIkG5tGNpnH47o1Y4/DO9dpd7WXd6lmXtnf0PHVmk8Mn9VlXMuO7Y3R/X2xgk6tU6jW7tmANw+6yveXv4Tx/bNZMX6/CoTAAB+c7A3VpFoxsXH9Kzx8ly91a8f3HSTlzD8mjWDF1+Ep5/2kog0SXWdQEYDH/mTB4BzLsfMPgDGUHMC8Y+OVr6JYBveHfXajLuRO7RH26DntExPZth+mTRPTeSvpx4Y9NLYPN/lsNc+X0dyovcrNHpQJ276ldfLSElMoHVG9QPz2wuKuPTZz8q+vmhoD/q0bwFA/04tGdi5VdCY671Zs6o/Nm5cxcQCrFifz/L13k2K7Zqlsm+rmmeBScNV1wlkADAjQPly4PQgdefi9VTuNrPxeJewDgWuAB7TGEjT1jItiZMHdeKOU4MPopc39pCuLP4hj2W3nhDxmIN/HKZ5ahJd22ZE9ByNgf8mxteXrOd131hRSlICSyaMICNF83Uao7r+qbYF8gKUbwXa1FTRObfHzI4GXsFLOH5PApdVV8/MLgYuBsjKCv/SiTQMy24dGVG9sdldGRvmeIxfjTOXmiD/jLQhWa255JhevPv1Rl5YtIbColJq6MRJAxaPfwsCrQEd9PKTmaUBLwLtgXP4uQcyASgGAk4Dcc5NBiYDZGdna/1pkRhJSkyokFR/rOZGR2k86jqB5OH1QiprQ+CeSXkXAsOB3s45/0jn/8xsOzDZzB5zzi2NWqQiUiu7fDeIvrBoDW0ykrlvzrds3VVIsm8SRFpyIv++6DAGdGoE40RNVF0nkOV44yCV9QdWBKl7AJBXLnn4feJ77AcogYjUEzN94yB3v/11hfLzj+rO1p17mf7pWlZv2R00gTjn+H7zLgp8CenNL35kyZptZQsBJyUkcP2JfenXMbx7kaT26jqBzATuM7OezrnvAcysO94U3euD1P0JaGNmvcvP4gIO8z2ui3awIhK5m37Vj3Oe+oQPrz+u7M2+fYs0EhOMr3/KZ/qnayucv313Ec53hXvawh/KFun8dHUea7ZWvRyW3a0NxaWOJWu2sDZvN+OO7gFAp1bpHNu3fQxbJn7m6nBbSt/NgEuBAuAmvPGQSUAL4EDn3E7fed2AVcBE59xEX1l3YBleIrkDbwwkG++mwm+BQ51zgW9F9snOznaLFy+OfsNEJCwr1ufzy4feo1/Hlpw4cF+e/2Q1P27fU+W8bu0yKHWOnXuKueGX/cpuKO3XsSVd22awIX8Ph905r0IdM/jy1pE0i2CZGanKzD51zgXcwKdOv8POuV1mdhzeUibP4A2ez8NbymRnuVMNSKTcjonOuVwzOxy4Fe/u9X2ANXgD5HcESx4iUn/kbvFm3X/1Yz5f/fjzrV23nNwf8N4ARg7cN+DyMuV1aJlG89Qk7jrtAA7t3pZ/f7KaB+eupLhU82XqQp2naN+aV6cFOSeXADOznHMrgLGxiUxE6sqI/h1olpLIf686hg4tvRsNE4yQFrSs7Mvbfp7CneCr/8qna2mZnsxD81aybffestUL0pMTmTruEHr7bvaU2lEfT0TqXHJiAssnjor6877+uTcUOvGNinNyzjm8G1t37WXWFz/y/aZdSiBRogQiIo3GraMHcO7Tn/C/a48tG7jv2CqNpMQEvly3nVlf/BjfABsZJRARaTSG7ZepFQLqUNVlTUVEREKgBCIiIhFRAhERkYgogYiISESUQEREJCJKICIiEhElEBERiYgSiIiIREQJREREIqIEIiIiEdFSJiLSpGwvKGL77iIAMlITy7bYlfApgYhIk7Ah39uw6tqXl3EtywBvY6q3rhgaz7AaNCUQEWkS9m3l7TsysHNLfn1QF97+8ie+/ik/SC2piRKIiDQJAzq1qrBS7+qtu5VAakkX/0REJCJKICIiEhElEBERiUhEYyBmti+QBaRVPuac+19tgxIRkfovrARiZp2BZ4FhgQ4DDkiMQlwiIlLPhdsDeRQYCPwZ+AIojHpEIiLSIISbQIYClzvnnolFMCIi0nCEO4heAGyMRSAiItKwhJtAngDOiUUgIiLSsIR7CWsdcI6ZvQO8CWytfIJz7uloBCYiIvVbuAnkMd9jd2B4gOMOqDGBmFlX4AFgBN7MrbnAlc651aEEYGb9gInAsUAzYDXwiHPu76HUFxGR6Ag3gfSozYuZWQbwDt7srfPwEs7twLtmdqBzbleQ+tm++vOB3wPbgT5A89rEJSIi4QsrgTjnfqjl610E9AT2d859B2Bmy4CVwCXA/dVVNLMEYCowzzl3arlD79YyJhERiUCkd6IPBI4B2gJbgP85574Moepo4CN/8gBwzuWY2QfAGGpIIHiXzPoDl0YSs4iIRFe4d6InAVOA3+GNX/g5M/s3cL5zrqSGpxgAzAhQvhw4PcjLH+17TDOzj4CDgTzgBeA651xB8BaIiEi0hDuN9xZgLDABbzwk3fc4ATjD91iTtnhv+pVtBdoEqdvJ9/giMAdvEP4evLGQf1dXycwuNrPFZrZ406ZNQV5CRERCFe4lrLOBSc65O8qV/QDcYWaJwAV4SaYmLkCZBSirzJ/snnXO+RPVfN/r3mVm/Z1zK6q8mHOTgckA2dnZgV5bREQiEG4PpBOwsJpjH/JzL6E6eXi9kMraELhnUt4W3+N/K5XP8T0ODlJfRESiKNwEsh44qppjR/qO12Q53jhIZf2BKr2HAHWhag/G33spDVJfRESiKNwE8hxwo5ndbGY9zSzdzHqY2V+AG4FgiyzOBA43s57+AjPrjpeUZgap+xbe/SOjKpWP9D0uDrENIiISBeGOgdyKdx/Hbb7P/Qx43ldekyeAy4AZZnYTXm9iErAGeLzsycy6AauAic65iQDOuS1m9lfgZjPLx7uhMBtv4H5q+anBIiISe+HeSFgMnGlmd+BtKtUWbwbVgkAD2AHq7zKz4/CWMnkGL/HMw1vKZGe5Uw1vY6rKPaSJwA7gD8A1wI/AvXhJSERE6pA513QmJmVnZ7vFi3WlS0Sg381vU1BUQlqy93+qYdw2ZgBjs7vGObL6xcw+dc5lBzoWtAdiZlnAj865It/nNQp1UUQRkXgqKPLueT7viO4APP1BDt/8tCOOETU8oVzCygGOAD4Bcgl8H0d52hNdROq9z24ewfptBQzs3AqA5z7W/77hCiWBjMMb0PZ/3nSueYlIo9W2WQptm6XEO4wGLWgCcc5NLff5lJhGIyIiDUa494FUYWb9zew0Mwt2F7qIiDQiYSUQM3vYzB4r9/WvgaXAdGCFmR0S5fhERKSeCrcHciLemld+twFvAIPwBtmDLaQoIiKNRLgJZF+8mViYWRe8da3+6pz7AngIUA9ERKSJCDeBFPDz/uPHAPn8vAbVTqBFlOISEZF6Lty1sD4D/mhmq4E/Av91zvlXwe2Bt7SIiIg0AeEmkBuBt/EGzrdRcX/yU/DGQUREpAkIdzHFRb7lTPoCK51z+eUOTwZWRjM4ERGpv8LtgeCc2wV8GqB8VlQiEhGRBiGUxRTPBWb59uM4N9j5zrlpUYlMRETqtVB6IFOAw/H2JJ8S5FwHKIGIiDQBoSSQ8rOresQwFhERaUBCWUzxh0Cfi4hI0xbuWliHm9nYao6dbmaHRScsERGp78K9E/2veMuXBNLPd1xERJqAcBPIIOCjao59AhxYu3BERKShCDeBpNVQJxFoVrtwRESkoQg3gXwFjK7m2Gjgm9qFIyIiDUW4d6I/BjxuZvnAE8BaoDNwMXAh8IfohiciIvVVuGthPWFm+wP/B1xV/hDwgHNucjSDExGR+iuStbCuMbNHgRFAW2AzMNc59320gxMRkfor7AQC4JxbBayKciwiItKAhDuIjpk1M7PLzexlM3vHzPr4yn9rZn2jH6KIiNRH4d6J3hVYBtwL9MHb1ta/je2xwDWhPIcv+Ww3s3wze9W3x0hYzOwvZubM7P1w64qISO2F2wP5G1CIlzwOBqzcsQXAsJoqm1kG8A7ehlTnAef4nutdMwv5HhIz64m3O+LGcIIXEZHoCXcMZARwsXNutZklVjq2Dm9Kb00uAnoC+zvnvgMws2V4OxleAtwfYhyPAs8B+xPhOI6IiNROuD2QFGBHNcdaAUVB6o8GPvInDwDnXA7wATAmlADM7ExgCPCXUM4XEZHYCDeBLANOq+bYiQTY6raSAcCXAcqXA/2DvbiZtQEeAP7snNsa7HwREYmdcC//3Au8bGYA//aV9TezMXh3ole3zIlfWyAvQPlWoE2Ir/8twXdGLGNmF+PdKU9WVthj9SIiUo2weiDOuVfxlis5HZjrK54GXAlc5px7O5SnCVBmAcoqnmA2FDgXGO+cC/QcgV/MucnOuWznXHZmZmao1UREJIiweiBm1gr4F/AMcATQHm+v9A+dc9WNjZSXh9cLqawNgXsm5T0OPAWsNbPWvrIkINH3dYFzrjCEGEREJApCTiBmloSXLE51zv2Hn3sg4VhO4A2p+gMrgtTt5/u4NMCxPLz1uR6MICYREYlAyAnEOVdsZhuAklq83kzgPjPr6V87y8y6A0cB1wepe2yAsgfx9iH5E/BdgOMiIhIj4c7Cehb4fS1e7wkgF5hhZmPMbDQwA1iDd4kKADPrZmbFZjbBX+acm1/5A9gGbPd9vbYWcYmISJjCnYWVC5xlZovw3vh/pNKguHPu6eoqO+d2mdlxeFNxn8EbPJ8HXOmc21nuVMPrWYS9VpeIiNSNcBPIP32PnfCWMqnMAdUmEADn3Gqqv5fEf04uIczMcs4ND3aOiIjERrgJ5DC8O9ELYhCLiIg0IEETiG/Nq5vx7vVogTeI/h/gQufctlgGJyIi9VcoPZBLgQnAfGAR3mKIpwL5wAUxi0xEROq1UBLIRcATzrlL/AVmdgnwsJld4pzbG7PoRESk3gplllNPYHqlshfxZkl1i3pEIiLSIISSQJrjXa4qz79sSQtERKRJCnUWVmffLoB+ieXKt5U/0X+HuYiING6hJpCXqyl/PUBZ5Z0KRUSkEQolgWimlYiIVBE0gTjnptZFICIi0rBorSkREYmIEoiIiERECURERCKiBCIiIhFRAhERkYgogYiISESUQEREJCJKICIiEhElEBERiYgSiIiIREQJREREIqIEIiIiEVECERGRiCiBiIhIRJRAREQkIkogIiISESUQERGJSJ0nEDPramYvm9l2M8s3s1fNLCuEetlmNtnMvjaz3Wa22syeM7MedRG3iIhUVKcJxMwygHeAvsB5wDlAH+BdM2sWpPpvgQHAQ8CJwPXAEGCxmXWNWdAiIhJQ0D3Ro+wioCewv3PuOwAzWwasBC4B7q+h7t3OuU3lC8zsAyDH97wTYhKxiIgEVNcJZDTwkT95ADjncnyJYAw1JJDKycNX9oOZbQI6xyJYEWk6dhYW89T7OazfVgBAq/RkbhszgNSkxDhHVn/V9RjIAODLAOXLgf7hPpmZ9QPaA1/VMi4REQBWbdrJkjXbeGHRGr7ftCve4dRrdd0DaQvkBSjfCrQJ54nMLAl4DNgEPFXDeRcDFwNkZQUdqxeRJmr6pUfQvV0zMluk8vA7K7lvzrec+Pf36NgqjcLiUgZ0askNv+wHQHKi0SuzOWYW56jjq64TCIALUBbJT+Fh4EjgJOdcoKTkvZhzk4HJANnZ2YFeW0SEQ7q3Lfu8Q8s0AAZ3bc1+HZrz0uK1vLdyMyf+/b2yc648vg+nHuRdPW+dnkKrjOS6DbgeqOsEkofXC6msDYF7JgGZ2V/xehXnOefmRCk2EREATs/uyunZP0/u3LW3hFnLfuSxs4ewaedebn79Sx6cu5IH564EoHlqEp/dPIKUpKZ1a11dJ5DleOMglfUHVoTyBGZ2I94U3sudc89EMTYRkYD+eeYQ/nmm93lxSSk3v/4lx/drzy8P6Mi732ziP0vXs7ekVAkkxmYC95lZT+fc9wBm1h04Ci8p1MjMLgduB250zv0jloGKiASSlJhA7l0nlX29Zede/rN0fRwjip+6TpdPALnADDMbY2ajgRnAGuBx/0lm1s3Mis1sQrmy3wIPAm8D75jZ4eU+wp7BJSIitVOnPRDn3C4zOw54AHgGb/B8HnClc25nuVMNSKRighvlKx/l+yhvATA8RmGLiEgAdT4Lyzm3GjgtyDm5VJqZ5Zw7Hzg/VnGJiEh4mtaIj4iIRI0SiIiIREQJREREIqIEIiIiEYnHUib1Wn5+Phs3bqSoqCjeoYg0GklJSaSlpZGZmUlaWlq8w5EoUQIpJz8/nw0bNtC5c2fS09Ob/EJpItHgnKO4uJidO3eyevVqOnToQKtWreIdlkSBEkg5GzdupHPnzmRkZMQ7FJFGw8xITk6mTZs2pKam8tNPPymBNBIaAymnqKiI9PT0eIch0milp6dTWFgY7zAkSpRAKtFlK5HY0d9X46IEIiIiEVECERGRiCiBNGJTpkzBzMo+WrRowaBBg3j44YcpLi4GYP78+ZgZ8+fPj1kc559/Pt27dy/7Ojc3FzNjypQpMXvNmvztb3/jwAMPxLmqG1S+//77mBkdOnQo+x5VVv57mpSURI8ePbjgggtYu3ZtrEOvEuuRRx5Jeno6++67L1dddRUFBQVB6w0fPrxCG8p/jBr18zql8+bN4+yzz6ZXr16kp6fTq1cvxo8fz8aNGys83+eff05GRgarV6+OehulftMsrCZg+vTpdOnShfz8fKZPn86f/vQnNm7cyMSJExkyZAgLFy6kf/+6WxG/Y8eOLFy4kF69etXZa/pt27aNO++8k8cffzzg9fipU6cC3oy8t956i5NPPjng85x//vlccsklFBcXs2TJEm655RY+/PBDlixZUicTMZYtW8aIESMYOXIkb7zxBjk5OVx77bWsW7eOF198sca6jzzyCPn5+RXKFi5cyFVXXcXo0aPLyh577DF27tzJTTfdRM+ePVm5ciW33HILs2fPZtmyZTRv3hyAgw46iBEjRnDzzTeXff+kiXDONZmPgw8+2NVkxYoVNR5vaP71r385wK1cubJC+fDhw12LFi3qLI7zzjvPdevWrc5eryb33Xef69ChgysuLq5ybPfu3a5Vq1Zu+PDhLiMjw5122mkBnwNwN954Y4WyKVOmOMC98sorMYm7slNOOcX17t3b7d27t6xs6tSpDnCffvpp2M83btw4l5KS4rZs2VJWtnHjxirnLViwwAHuqaeeqlA+a9Ysl5SU5NatWxf0tRrb39nkBatct+vecHm7Ct3e4hK3t7gk3iFFFbDYVfOeqktYTdAhhxzCjh072LhxY8BLWMOHD+foo49mxowZDBw4kNTUVPr27ctLL71U5bmWLl3K6NGjadOmDenp6Rx11FG89957Nb5+oEtY559/Pl26dOHzzz9n6NChZGRk0KdPHx577LEq9XNycjjrrLPIzMwkNTWVwYMH89prr4XU9ieffJIzzjiDxMTEKsdef/11tm/fzh/+8AdOPfVU3njjDfLy8kJ63kMOOQSA7777LqTza6OoqIi3336bsWPHkpycXFY+duxYUlJSmDFjRljPV1BQwPTp0zn55JNp27ZtWXlmZmaVc/3tXLduXYXyE044gZYtW8btsmQ8zVnxEwCDJ/6XPje+RZ8b3+Ket7+Oc1R1QwmkCcrJySExMbHsEkQg3333HZdffjlXX301r776Kr179+a3v/0t7777btk5n332GUceeSRbt27liSee4JVXXqFdu3Ycf/zxfPrpp2HHlZ+fz5lnnsnZZ5/NjBkzOOSQQxg/fnyF11yzZg2HHXYYS5cu5YEHHmDmzJkMGTKE0047jZkzZ9b4/KtXr+brr79m6NChAY9PnTqV1q1bM3r0aM4991wKCwt54YUXQoo9JycHgNatW9d4XnFxcdCPkpKSGp9j1apV7Nmzh4EDB1YoT0tLo1evXqxYsSKkmP1effVVduzYwXnnnRf03AULFgDQr1+/CuVJSUkcccQRvP3222G9dmMwfrh3KfbqEftxzQn7AfDI/FWMefh9xjz8Pmc8vpD124KPTTVEGgMJ4rb/LGfF+vzgJ8ZQ/04tueXkARHXLykpobi4mB07dvDSSy/x6quvcvLJJ9d4x/2GDRtYuHAhhx9+OACjRo1iwIABTJgwoayHce2115KVlcU777xDSkoKACNHjmTgwIFMmjSJ119/Paw4d+zYwSOPPMKxxx4LwLBhw5gzZw7PP/98Wdmtt96Kc44FCxbQrl27stdcs2YNEyZMqHANv7KFCxcCMGjQoCrH1q9fz9y5c7nwwgtJTU3l+OOPp3PnzkydOpXx48dXOd/5lufwj4Fcc801ZGRk8Ktf/ara18/NzaVHjx5Bvw/dunUjNze32uNbt24FoE2bNlWOtW3btux4qKZNm0b79u058cQTazxvx44dXHnllfTr149TTjmlyvGDDjqIe++9l9LSUhISms7/psf17VBhj/T75nwLQJtmKeQXFPFxzla++jGfTq0b303KSiBNQN++fcs+T0hI4KyzzuLBBx+ssU7Xrl3LkgdAYmIip59+Ovfccw+lpaUUFhayYMECbrjhBhISEirMWDr++ON57rnnwo4zIyOjLFEApKam0qdPnwqze95++21++ctf0qpVqwqvOXLkSK699lry8/Np2bJlwOdfv349EPjSzLPPPktJSQnnnnsu4H2fzj77bO6++26++eYb9t9//wrn33nnndx5551lXx9wwAG8+eabdOrUqdr2derUiUWLFtX0LShrd02cb/ZYoEkA/mOh8ifOK664gqSk6t8OiouL+d3vfse6dev44IMPAp6bmZlJYWEhW7duZZ999gkrjsakfDJZtnYbox/+II7RxJYSSBC1+c+/vnjttdfo0qULLVq0oFu3biGthtqhQ4eAZXv37mXTpk1ll1omTZrEpEmTAj5HuP+JBvqPOjU1lT179pR9vXHjRqZNm8a0adMCPseWLVuqTSD+5wn0Bj1t2jSysrIYMGAA27ZtA2DMmDHcfffdTJs2jTvuuKPC+ePGjWP8+PEkJSXRtWvXst5QTVJSUhg8eHDQ84Ldre0fpwjU08jLy2PAgNB/Z5999llKS0trvHzlPz537lxmzZrFgQceGPA8/+yzUKYSS+OgBNIEDBw4kN69e4dVZ8OGDQHLUlJSyMzMpKCggISEBP74xz+W/ddeWSwuY7Rr146hQ4dy3XXXBTxeUw/A/yafl5dXYart4sWLWb58ORA4iT3zzDNMmjSpQns6duxIdnZ2WLFH6xJWr169SE1NLYvZb8+ePXz//fecfvrpIcc0bdo0Bg0aFPCynt+ll17Kiy++yMsvv8wvfvGLas/zJ7Sm3PuobMcer5d84dTFDO7ami27CinYW8KNJ3ljSDv3FNO/U0taZ3iXgFukJdG+RcNZ7l4JRAJas2YNH330UdllrJKSEqZPn86hhx5KQkICzZo1Y+jQoSxdupQhQ4bU2TXvUaNGsXDhQgYMGBD2/Rb+S3nff/99hUQzdepUzIyXX365wiwkgNmzZ3PXXXcxf/58jjvuuFrFHq1LWCkpKYwaNYqXXnqJW2+9texy0ssvv0xhYWGN40Dl+RPn/fffX+05V199NU8++SRTp04NOO5RXk5ODl27dtWCpOV0a+eNM+7TPJWW6cksWbMNgP97cWnA8xMMFv7lF3Ro2TCSiBKIBNShQwfOOOMMbrvtNjIzM3n00Uf59ttvefTRR8vOuf/++xk2bBgjR47kwgsvpGPHjmzevJnPPvuMkpIS7rrrrqjHNXHiRA499FCGDRvGZZddRvfu3cnLy+PLL7/k+++/5+mnn6627qGHHkpqaiqffPIJRx99NOBNiX3hhRc45phj+PWvf12lzuDBg3nwwQeZOnVqrRNISkpK2L2W6tx6660cccQRjB07lj/+8Y/k5uZy7bXX8pvf/IaDDz647Lxp06Yxbtw45s2bxzHHHFPhOaZNm0ZSUhJnnnlmwNe4++67uf/++xk3bhx9+vTho48+KjuWmZlZ5UbQjz/+mGHDhkWlfY1FlzYZFcZENu7Yw2c/bKPvvi0AuG/ON+zbMo0DurTi89XbmPJhLtsLipRApGHr3bs3f/7zn7nhhhtYuXIl3bt3rzAbCmDIkCEsWrSI2267jcsvv5zt27eTmZnJkCFDuPTSS2MSV1ZWFosXL+bWW2/lhhtuYNOmTbRr146BAwcGnYaalpbGmDFj+M9//sNVV10FwBtvvMHmzZsZN25cwDqtW7fm17/+Na+88gr//Oc/a5z6XJcGDx7M7Nmzue666zjppJNo1aoV5557boWBffDGL0pKSqoMrhcVFfH8888zatSogONdAG+99RYATz/9dJXEfN5551W452PNmjUsXbq02vEw8bRvkcaogfuWff3wmUPKPk9KSGDKh7lcOHURfdp7CWZE/w787tCsOo8zVBburI2GLDs72y1evLja41999VWV+e1N0fDhwykuLub999+PdyhR578UlZubS1ZW/f3DbGjuvvtuHn30UVatWhXwJs3y9HcW2A9bdnHN9KUUFJVgGLmbd9EzsxkzLjs6rnGZ2afOuYBdZ/VApEkZPnw4xx9/PPfccw8PP/xwvMNpFPbs2cPf//537rrrrqDJQ6rXrV0zpl96ZNnX5//rE/J27Y1jRME1nbt9RHweeughunTpEvY9ExJYbm4uV1xxBeecc068Q5E6Vuc9EDPrCjwAjAAMmAtc6ZwLuha0maUBk4CzgdbAEuA659z/YhVvUxTLpd3rg759+3L99dfHO4xGo2/fvhVuVpWmo057IGaWAbwD9AXOA84B+gDvmlmzEJ7iKeAiYALwK+BHYLaZDY5JwCIiUq267oFcBPQE9nfOfQdgZsuAlcAlQLUT0s1sEHAmMM459y9f2QJgOTARCG3yu4hIA5BgxtK129n/prdITDASzUhIMBITDAPat0xj6rhDSPCtXNCuWUqd7zlf1wlkNPCRP3kAOOdyzOwDYAw1JBBf3SKgbLcc51yxmb0AXG9mqc65wtoG6Jyr8x+CSFOhcafQXXZcb/bftwWlpY6SUkeJc97nzvHvj1ezZddeDr1jXoU6rdKTSUrwJRrzkk1igtGueQqv/eGoqMdY1wlkABBos4LlQLD1FwYAOc653QHqpgC9fZ9HLDk5mYKCghpXqRWRyBUUFAS90148Q7LaMCSr6tI6ANecsD9vfvETJaWlACxfn096SmJZginxJ51SKHWOjJTYzI6r6wTSFgi0Q89WIPB3KrS6/uO10r59e9atW0fnzp1JT09XT0QkCvxL3+/YsYPNmzdXe+OihK51RgpnHhb/+5jicR9IoD5sKO/UFkldM7sYuBgIeuOYfxXX9evXU1RUFEJIIhKKpKQk0tLSyMrKCmk1aGkY6jqB5BG4p9CGwL2L8rYCgTJAm3LHq3DOTQYmg3cnerAAW7ZsWe1y4CIi8rO6vpFwOd5YRmX9gWD7cC4HevimAleuuxeI/WbUIiJSpq4TyEzgcDPr6S8ws+7AUb5jweomU26w3cySgDOAOdGYgSUiIqGr6wTyBJALzDCzMWY2Gm9W1hrgcf9JZtbNzIrNbIK/zDm3BG8K74Nm9nsz+wXwAtADuKXumiAiIlDHCcQ5tws4DvgWeAZ4DsgBjnPO7Sx3qgGJAeK7APgXcDswC+gKjHLOfRbj0EVEpJI6n4XlW/PqtCDn5BJgdpVzrgC4yvchIiJxpNV4RUQkIk1qQykz2wT8EGH1fYDNUQynIVCbG7+m1l5Qm8PVzTmXGehAk0ogtWFmi6vblauxUpsbv6bWXlCbo0mXsEREJCJKICIiEhElkNBNjncAcaA2N35Nrb2gNkeNxkBERCQi6oGIiEhElEBERCQiTTqBmFlXM3vZzLabWb6ZvWpmIe3SYmZpZnavmf1oZgVmttDMhsU65tqKtM1mlm1mk83sazPbbWarzew5M+tRF3HXRm1+zpWe5y9m5szs/VjEGU21bbOZ9TOz6Wa22ff7/Y2ZXRHLmGurln/PWWY21fd7vdvMvjWz282sWazjjpSZdTGzf/jee3b7fje7h1g3Ku9fTTaB+JaFfwfoC5wHnAP0Ad4N8ZfmKeAiYALwK+BHYLaZDY5JwFFQyzb/Fm8p/oeAE4HrgSHAYjPrGrOgaykKP2f/8/QEbgQ2xiLOaKptm80sG/gYSAV+D/wS+Bve+nT1Um3a7Ds+FxgG3AycBDwJXA08HcOwa6s3MBZvL6X3wqwbnfcv51yT/ACuAEqA3uXKegDFwFVB6g7C2x3xgnJlScA3wMx4ty1Gbc4MUNYNKAUmxrttsWhzpeeZjbdi9Hzg/Xi3K4Y/5wS8vXdei3c76rDNJ/j+nk+oVH6Xr35GvNtX3c+q3Oe/97Whewj1ovb+1WR7IMBo4CPnXNlGVM65HOADYEwIdYvwlpf31y3GW15+pJmlRj/cqIi4zc65TQHKfgA2AZ2jHGc01ebnDICZnYnX2/pLTCKMvtq0eTjeJm33xyy62KhNm1N8j/mVyrfhJdRQttyuc8650girRu39qyknkAHAlwHKl+P9AQWrm+Oc2x2gbgpe17I+qk2bqzCzfkB74KtaxhVLtWqzmbUBHgD+7JwLuG1yPVSbNh/te0wzs4/MrMjMNprZQ2aWHtUoo6s2bZ4LrATuNrP+ZtbczI7D69U85rxtKBqTqL1/NeUE0pbA+7Bv5ed91iOp6z9eH9WmzRX4doN8DK8H8lTtQ4uZ2rb5Xrz9a6ZEMaZYq02bO/keXwTmACOAe/Aukfw7WgHGQMRtds7twUuc/st3O4B5wBvAZdENs16I2vtXne8HUs8EuosylO6q1aJuvEUr7oeBI4GTnHOBfhnrk4jabGZDgXOBIc53obgBifTn7P+n8lnnnH9H0PlmlgjcZWb9nXMrohJh9EX6c07DS5jt8QbfVwOH4g0wFwPjoxhjfRC196+mnEDyCJxp2xA4O5e3FQg0PbBNueP1UW3aXMbM/gpcDJznnJsTpdhipTZtfhyvd7XWzFr7ypKARN/XBc65wijFGU21afMW3+N/K5XPwRtUHgzUxwRSmzZfiDf209s5t8pX9j8z2w5MNrPHnHNLoxZp/EXt/aspX8JajnctsLL+BP8DWQ708E0drFx3L/Bd1Sr1Qm3aDICZ3Yg3hfcK59wzUYwtVmrT5n7ApXhvQP6Po4DDfZ/X1/9Ma/u7DVX/Q/X/dxrpwG2s1abNBwB55ZKH3ye+x361jK2+idr7V1NOIDOBw33z+wHw3YRzlO9YsLrJwOnl6iYBZwBz6ul/pVC7NmNml+PtR3+jc+4fsQoyymrT5mMDfCzFG6w9Fng5BvFGQ23a/BZQCIyqVD7S97g4SjFGW23a/BPQxswqDx4f5ntcF60g64novX/Fey5zHOdQN8PLtF/gTfMbjffm8D3QvNx53fCug06oVP8FvP9Cfw/8Au/NZA/e9fK4ty/abca7kbAU7w3m8Eof/ePdtlj9nAM833zq/30gtf3dvsVXfidwPF6PswCYEu+2xaLNQHe8Kbzf4t2EeCxwra9sMeXut6hvH8BvfB+P4vUax/u+PibIzzgq719x/wbE+ZufBbzi+0XZAbxOpRtxfL9cDri1Unk63lz5n3zf+I+B4fFuU6zajDcLyVXzMT/e7YrVzznAc9X7BFLbNuNdrrrK94a8F28b6IlAcrzbFcM29wdeAtbgJctvgfuANvFuV5A21/g3Gev3Ly3nLiIiEWnKYyAiIlILSiAiIhIRJRAREYmIEoiIiERECURERCKiBCIiIhFRAhEJg5md79s61P+x18xWmdmdvkX54hXXFDPLLfd1d19858crJmn8mvJiiiK1cTqwFmgBnIq32VQL4E/xDEqkLimBiERmift597v/mlkf4EIzu8JFvlOcSIOiS1gi0fEZ3vIQ+wCYWYaZ3W1mOb7LXDlmdqOZVfibM7NMM3vEzNaYWaHv8Rn/tqJm1tv3dY6ZFZjZ92b2qG+nRJG4Ug9EJDq6A9uBLb6VTWfjra80CW+Bv8OBm/H2rLgayrbL/dBXdjuwDG9TozF4W4sW4u0QuBa4Em/xu57ADcCbwBF10TCR6iiBiEQm0Zco/GMgpwFXOudKzOwcvC1Sj3HO/c93/jwzA7jFzO52zm0E/g8vIWQ75z4v99zP+z/x1fc/B2b2Id4ih++Z2UGV6onUKV3CEonM10AR3u5tTwGPO+ce9h0bhbeC7YdmluT/wNvVLxmvNwJwArCopiRgZilmdoOZfW1mBb7XfM93eP+ot0okDOqBiETmVLxLS5l4S5//wcw+ds5Nw7sM1Q3vzT6QduUeg22V+le8mV0T8S537QC6AK8CcZs2LAJKICKR+tI/C8vM3sEbv7jXzF7B21c8BxhbTd1c3+NmoHOQ1/ktMM05d7u/wMya1yJukahRAhGpJedcoZldC8wA/gC8jTcmstM593UNVecAN5nZIOdcdT2RDKr2ZC6obcwi0aAEIhIFzrmZZrYIuAbojfcmP8/M/oZ3mSoF6IW31eopzrndwAPAmcBcM7sdb7bWPnizsC51zu3AS0bnmdkXeIPnvwaOrNPGiVRDCUQkem7Cm777e2Ak3l7iFwM9gF3AKmAW3jaxOOe2mdlReFN4r8cbE9kAvOM/B2/8w4A7fF+/CfwO+CT2zRGpmba0FRGRiGgar4iIREQJREREIqIEIiIiEVECERGRiCiBiIhIRJRAREQkIkogIiISESUQERGJyP8DcWakID9LiYMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_precision_recall_curve(pipe_lr, X_valid_fold, y_valid_fold);\n",
    "plt.plot(recall_score(y_valid_fold, pipe_lr.predict(X_valid_fold)), precision_score(y_valid_fold, pipe_lr.predict(X_valid_fold)), '*r', markersize=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The threshold is not shown here, but it's going from 1 (upper-left) too 0 (lower right).\n",
    "- The red star is the threshold 0.5.\n",
    "- It's bumpy because changing the threshold affects some discrete number of examples.\n",
    "- The top-right would be a perfect classifier (precision = recall = 1).\n",
    "- It seems like there might be some good compromises here at precision = 0.7-0.8. \n",
    "- In the plot, \"AP\" is a score that summarizes the \"goodness\" of the above plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import average_precision_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7229898256547778"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_precision_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AP vs. F1-score\n",
    "\n",
    "It is very important to note this distinction:\n",
    "\n",
    "- **F1 score is for a given threshold** and measures the quality of `predict`.\n",
    "- **AP score is a summary across thresholds** and measures the quality of `predict_proba`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Why is the precision-recall curve not monotonic? \n",
    "\n",
    "From last year's Piazza:\n",
    "\n",
    "Q: Is this right: When we lower our threshold we have a higher recall (because we are predicting more positives) but we also have a lower precision (because we are increasing the number of FNs)?\n",
    "\n",
    "A: The only that that is guaranteed is that when you lower the threshold you will predict as positive all those instances you were previously predicting as positive, plus possibly more. Recall is TP/positives. The number of positives is unaffected by your threshold, so if TP goes up then recall goes up. Thus, when you decrease the threshold, your recall might stay the same or it might go up. However, the precision is another story - it could actually go down. Precision is defined as TP/(TP+FP). If your model produced really bad probability scores, it might be the case that lowering the threshold would cause you to predict more positives, but the true class for all those cases was negative. Thus TP would stay the same and FP would go up, so precision would go down."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)\n",
    "\n",
    "<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ROC curves (10 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The [ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) is very similar to the precision-recall curve, but with different axes\n",
    "- ROC plots true positive rate (aka recall) ($\\frac{TP}{TP + FN}$) against false positive rate ($\\frac{FP}{\\# negatives }$) \n",
    "- Note that the \"C\" doesn't stand for \"curve\", so saying \"ROC curve\" isn't like saying \"PIN number\" or \"SIN number\".\n",
    "- The diagonal line is what happens if your guesses are random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import plot_roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_valid_fold, pipe_lr.predict(X_valid_fold))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEQCAYAAACeDyIUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0EElEQVR4nO3deXxU1f3/8deHLRAVWQREtmixKlC1FJeKQkQqtlbQurVuqLhb9y9W66+o4Nq6fW2tSq0KaqtVUKi2LqjgUsCiFb8GNzAIBRSQCCgpsnx+f5ybdDKZTG5myWLez8djHpO59557P3eSzGfOPeeeY+6OiIhIXbVo6ABERKRpUgIREZGMKIGIiEhGlEBERCQjSiAiIpKRVg0dQH3aYYcdvKioqKHDEBFpMt58883V7t4l1bpmlUCKioqYN29eQ4chItJkmNknNa3TJSwREcmIEoiIiGRECURERDJS7wnEzHqa2W/NbLaZbTAzN7OimGXbmtlvzGyFmZVH+xiS55BFRCSFhqiB9AWOA8qAV+tY9o/AmcA44MfACuA5M9s7lwGKiEjtGqIX1ivu3g3AzM4ADo1TyMz2Ak4ATnf3B6Jls4ASYDwwMj/hiohIKvVeA3H3rRkWHQlsAh5L2Ndm4FFghJkV5CA8ERGJqSndB9IfKHX3DUnLS4A2hEtjJfUe1TfMn+YuYdrbyxo6DBHJoX47tefqI/rnfL9NqRdWJ0K7SbI1CeurMbOzzGyemc1btWpV3oL7ppj29jIWrFjX0GGISBPQlGogBqSa/crSFXL3icBEgEGDBmn2rBj6dW/PY2d/v6HDEJFGriklkDVA7xTLOyaslwwkXrZasGId/bq3b+CIRKQpaEqXsEqAnc2sMGl5P+BrYGH9h/TNkHjZql/39ozau0cDRyQiTUFTqoFMB64FjgUmAZhZK+B44Hl339iAsTUZqRrJK2odumwlInXRIAnEzI6Jfvxe9PxDM1sFrHL3WWbWB1gEjHf38QDu/raZPQbcYWatgVLgXGBn4MT6PYOmq6K2kXiZSrUOEclEQ9VAHk96/fvoeRZQTGgYb0n1S2ynAdcD1wEdgPnAYe7+Vr4CbYyy6Wqr2oaI5EqDJBB3r63n1GJS9K5y93Lg0ujRbKWqRcSl2oaI5EpTagORBKpFiEhDUwJpQiouXamrrYg0Bk2pG2+zl5g8dBlKRBqaaiA5ls+xpNQALiKNiWogOZbPsaRU8xCRxkQ1kBxINRSIagki8k2nGkgOaCgQEWmOVAPJEdU6RKS5UQ1EREQyogQiIiIZUQLJ0p/mLmFuqaYiEZHmRwkkSxW9r9RwLiLNjRrRM5Q4rMh+O3fihP1STZYoIvLNpRpIhjSsiIg0d6qBZEFdd0WkOVMNREREMqIEIiIiGdElrDpINeaViEhzpRpIHWjMKxGR/1INpI7UcC4iEqgGIiIiGalzAjGzbc2sj5m1zkdAIiLSNMROIGb2YzN7C1gLLAK+Ey2/z8xOyFN8IiLSSMVKIGZ2JDANWA38IqlcKTA655GJiEijFrcGcjXwgLsfCtyRtO5dYEAugxIRkcYvbgLZA3gs+tmT1pUBnXMWkYiINAlxE8g6YIca1hUBq3ISjYiINBlxE8gLwJVm1iFhmZtZAfBz4O9xD2hmvczsCTNba2brzGyqmcUaC93MepvZJDNbYmYbzOxDM7vOzLaJe/xM/GnuEo6/d3blTYQiIhL/RsKrgDeAD4C/ES5jXQHsCWwPHBlnJ2ZWCLwEbCQ0vDtwHfCyme3p7l+lKbsNMANoDfwKWALsA1wL7AocH/Nc6kxDt4uIVBcrgbj7YjMbSPiwHgFsAYYAzwLj3H15zOOdCewC7ObuCwHM7B3gI+Bs4LY0ZQcTEsUId38+WvaymXUC/sfMCt19Q8w46kx3oIuIVBV7KBN3/zcwJsvjjQTmVCSPaL+lZvY6MIr0CaRN9Jx8HekLwqU4yzI2ERGpg7j3gbxkZrvXsO7bZvZSzOP1J3T7TVYC9Kul7AxCTeVmM+sX3RE/DLgIuCfd5S8REcm9uI3oxUBNY5dvBwyNuZ9OhG6/ydYAHdMVdPf/AAcSYi4B1gMvAk8TGvJTMrOzzGyemc1btUqdxUREcqUuY2El3/9R4VvAl1nup9bLT2bWlnAvSlfgZELSGktoPL+rxoO5T3T3Qe4+qEuXLnUIU0RE0qmxDcTMTgNOi146MNHM1idt1o5wF/qLMY9XRqiFJOtI6ppJojGEmlBfd18ULXvFzNZGsd3j7vNjxiEiIllKVwPZSuhttYVQQ0h8XfH4HLib+I3rJYR2kGT9gAW1lP0OUJaQPCq8ET3vETMGERHJgRprIO4+CZgEYGYvA+e6+/tZHm86cIuZ7eLuH0f7LiJ00b2ilrKfAh3NrG9iLy5gv+h5WZaxiYhIHcRqA3H3g3OQPAD+ACwGppnZKDMbSRjldylwb8VG0Xwjm81sXELZBwkN538zs9FmdrCZjQVuAd4EXs9BfCIiElOdprQ1s72A3YC2yevcfXJt5d39q6jr7e3AQ4RLYy8CF7t7YkO8AS1JSHDRzYz7A9cQ7l7fgZB4JgLXu/vWupyLiIhkJ1YCicbAegbYv2JR9JzYo6rWBALg7kuAo2vZZjEpema5+wLguDjHERGR/IrbjfcGwpDtQwgf7EcBw4BHgI+BffMSnYiINFpxE8gIQhKZE73+t7vPdPdTCHeIX5SP4EREpPGKm0C6Ax+7+xbgP4S7zytMBQ7PdWAiItK4xU0gnwIdop8/ARKHpe2by4BERKRpiNsL6zVC0nia0Hvq6uj+jc2EeT2m5yU6ERFptOImkGuBnaKff0NoUD8eKCQkjwtyH5qIiDRmcSeUWgQsin7eBFwWPUREpJmqy2i8KZnZd83syVwEIyIiTUfaGoiZtQS+B/QGFrn7vxLWDQKuBn5EGGJERESakRprIGbWE5gLzAb+Aswzs8fMrI2Z3RetGwbcSpjnXEREmpF0NZCbgN2BXwFvATsDvyQMWvg9wki9V7j7Z/kOUkREGp90CeQQ4Bp3v6VigZl9QLjz/LfurrvPRUSasXSN6F3479AlFWZHz4/nJxwREWkq0iWQFsDXScsqXm/ITzgiItJU1HYfyBFmNiDhdQvCEO4jzWzvxA3d/f4cxyYiIo1YbQnkqhqWj0t67YASiIhIM5Iugexcb1E0ctt+uZYL7r8GjnkGOndu6HBERBqFGhOIu39Sn4E0ZkNnP8NeC+bCpElw6aUNHY6ISKOQ9VAm33juHP7iY2F+3dtvB/faSoiINAtKILV59VUKy78MP3/xBbz2WoOGIyLSWCiB1OaOOyj4+j/h56++CrUQERFRAqli1Cgwq/p45hlaVFy2codnnqm+zahRDRu3iEgDUAJJdMMN0Ls3tG3732VfJ91Lmfi6bVvo0yeUExFpZuqUQMyshZkNMLOhZrZNvoJqMP37w4IFMHIkFBam37awMNQ8SkpCORGRZiZ2AjGz84FPgfnAS8Bu0fKnzOzC/ITXALbZBh57DG69FQoKUm9TUBDWP/po2F5EpBmKlUDM7Ezgf4GnCHOhW8LqV4Gjcx5ZQxs4MH0C+d736jceEZFGJm4N5FLgVnc/C0ievvZ9otpIHGbWy8yeMLO1ZrbOzKaaWe86lN/DzB43s9VmVm5mH5hZ7oeWnzcPNm0CYCvGxjYFocEcwvJ583J+SBGRpiRuAtkZeK6GdV8BHeLsxMwKCZe/dgdGAycDuwIvx2lTiabRnQsUAGcQptO9FWgZ5/h18uqrUF4ObduyulM37jz9GujVKzScl5eH9SIizVhtgylWWA0U1bBuN2BZzP2cSZj+djd3XwhgZu8AHwFnA7fVVNDMWhBmQXzR3Y9KWPVyzGPXzdy50LIljBrF/xxwFhsL2sEtF8Dpp8OUKWG9iEgzFrcG8ldgnJklzn3uZrYDcAmhbSSOkcCciuQB4O6lhGlya7uZohjoR5okk1N77AETJ8Kjj4bkAf9tYJ84EXbfvV7CEBFprOImkP8HbATeJUxp68CdwHvAFmB8zP30j/aRrISQHNI5MHpua2ZzzGyTma00szvNrF3M48f3zDOhtpHK6aeH9SIizVisBOLunwODgBuB1sAiwuWv3wHfd/e1MY/XCShLsXwN0LGWsjtFz48BzwM/AH5NaAv5U02FzOwsM5tnZvNWrVoVM0wREalN3DYQ3H09MCF6ZCPVcLaWYlmyimT3sLtXTGg108xaAjeZWT93X1DtYO4TgYkAgwYN0lC6IiI5Evc+kNuSp7DNUBmhFpKsI6lrJok+j55fSFr+fPS8d+ZhiYhIXcVtAzkNeNPM3jWzsWbWI8PjlRDaQZL1A6rVHlKUheo1mIray9YMYxIRkQzETSDdgOOAhYRLWJ+Y2QwzO7mOY2JNB/ZP7M1lZkXA4GhdOn8nNOQflrR8RPSsO/tEROpR3Eb0r919irsfCXQHLgTaEe7L+MzMHop5vD8Ai4FpZjbKzEYC04ClwL0VG5lZHzPbbGYVbR0VDfk3AueY2Q1mNtzMrgDGAZMSuwaLiEj+1Xk4d3cvc/ffu/tg4GBC28UJMct+BQwDPgQeAh4BSoFh7v5lwqZGuLs8Ob7xwOWE2tDfgHOB3xBuUBQRkXoUuxdWheiS1THASYSb+zYDU+KWd/cl1DL4orsvJkXPLHd3wo2E9XMzoYiI1ChuL6wWZnaYmT0CfAbcTxiP6jxgR3c/Lo8xiohIIxS3BrIc6EJoRL8ZeCiqJYiISDMVN4FMASa7u0YQFBERIGYCcffz8x2IiIg0LTUmEDMbArzl7l9GP6fl7q/kNDIREWnU0tVAZgL7A29EP9c0jpRF63I/qZOIiDRa6RLIwfx3eJFh1JxARESkGaoxgbj7rISfZ9ZLNCIi0mTEvQ/kYzPbq4Z1A8zs49yGJSIijV3coUyKCDcOptIW6JOTaEREpMmoy1hYNbWBDAK+yD4UERFpStJ1470EuCR66cBfzezrpM3aESaIejQ/4YmISGOVrhfWx8CL0c+jCfNtJE8qvpHQU+u+3IcmIiKNWbpeWNMIc3VgZgDj3b20nuISEZFGLu5QJqflOxAREWla0rWBjAPuc/fliTMD1sDdfUJuQxMRkcYsXQ3kGuBZwlDu19SyHyfMlS4iIs1EujaQFql+FhERgQzmRBcREYH4Q5l828z2TXjdzsxuNLO/mtnP8xeeiIg0VnFrIL8Djkl4fT1wGbATcLuZacIpEZFmJm4C2RN4HcDMWgCnAL9w9+8B1wFn5Sc8ERFprOImkA7A59HP3wU6Ak9Er2cCu+Q0KhERafTiJpDPgL7Rz4cCi9x9afR6W2BzrgMTEZHGLdad6MB04EYzGwCcCtybsO47hHGzRESkGYmbQK4gzPsxgpBMbkhYNxJ4PsdxiYhIIxd3LKyvgDNrWHdATiMSEZEmoU43EppZJzM73MxONrMfmVmnuh7QzHqZ2RNmttbM1pnZVDPrncF+rjQzN7PX6lpWRESyF/cSFmZ2HeHej8SpbTea2S3u/quY+ygEXiLMIzKaMIbWdcDLZrZnVNOJs59dgKuAlXHjFxGR3IqVQMzsYuCXwB+Bh4FPgR2Bk4Bfmtkqd78zxq7OJHT53c3dF0b7fgf4CDgbuC1m3HcDjwC7xT0HERHJrbiXsM4B/tfdz3T3We7+QfR8JnAncF7M/YwE5lQkD4BokqrXgVFxdmBmJwADgStjHlNERPIgbgIpAp6pYd0z0fo4+gPvplheAvSrrbCZdQRuBy539zUxjykiInkQN4F8DgyoYV1//nuXem06AWUplq8h3N1em98AHwIPxjweZnaWmc0zs3mrViVP6S4iIpmKm0CeBCZEva9aA5hZKzP7GTAemFKHY3qKZVZbITM7iDAG17nunmofqQ/mPtHdB7n7oC5dutQhTBERSSduArkSeBuYBGwws8+AckJD9nxCA3scZYRaSLKOpK6ZJLqX0Ij/bzPrYGYdCA3oLaPXBWlLi4hITsW9kXC9mQ0BDgcOIiSBNcAs4O91qBGUEC55JesHLKil7B7R45wU68qAS4A7YsYhIiJZSptAzGwHQlfdvoQP6Snu/ossjjcduMXMdnH3j6NjFAGDCcOlpHNwimV3AC2BC4CFKdaLiEie1JhAzGw34BUgseHgCjM7xt2nZXi8PwA/B6aZ2f8jtIdMAJaSMECjmfUBFgHj3X08gLvPTBHjF0CrVOtERCS/0rWBXAf8BygGtiGMuvsG8W/2qya603wYoSfVQ4Q2lFJgmLt/mbCpEWoWmrNdRKSRSncJaz/gV+7+SvS6xMzOBuabWRd3z6hPrLsvAY6uZZvFxOiZ5e7FmcQgIiLZS/cNvwfwQdKyDwgf7DvlLSIREWkS0iUQA7YkLdsao5yIiDQDtXXjvdbMVie8rrisNMHMEocScXcfndvQRESkMUuXQJYQ7rtI9gnV7+WIfWe4iIh8M9SYQNy9qB7jEBGRJkZtGSIikhElEBERyYgSiIiIZEQJREREMqIEIiIiGVECERGRjMSaD6SCme0JDAE6A/e6+6dm1hf4zN3X5yNAERFpnGIlkGi2v4eBnxDuRnfgr8CnwK8Jo+vWNp+HiIh8g8S9hHU9MBw4GehG1ZFy/w6MyHFcIiLSyMW9hPUz4P+5+5/MrGXSulKgKKdRiYhIoxe3BtIZeC/NPgpyE46IiDQVcRNIKfD9GtbtS/V5Q0RE5BsubgKZTJgP/USgTbTMzexg4BLg/nwEJyIijVfcBPJr4BnCPOYV84C8BswAnnX33+YhNhERacRiNaK7+xbgp2Z2F6HHVVfgc0LymJXH+EREpJGq042E7v4q8GqeYhERkSZEQ5mIiEhG4t6JvpVapq119+T7Q0RE5Bss7iWs8VRPIJ2BQwn3gDyYw5hERKQJiNuIfk2q5dFd6X8F1uYwJhERaQKyagOJemf9Hrg4J9GIiEiTkYtG9AKgU9yNzayXmT1hZmvNbJ2ZTTWz3jHKDTKziWb2vpltMLMlZvaIme2cVfQiIpKRuI3oqT7g2wADgJuAeTH3Uwi8BGwERhPaVa4DXjazPd39qzTFfwr0B+4ESoAewK+AeWa2t7svjRODiIjkRtxG9MWk7oVlwCLg/Jj7ORPYBdjN3RcCmNk7wEfA2cBtacre7O6rqhzc7HXCOF1nAuNixiAiIjkQN4GclmLZf4BPgH9GbSFxjATmVCQPAHcvjRLBKNIkkOTkES37xMxWEWojIiJSj2pNIFFPq7eB5ak+xOuoPzAtxfIS4Ni67szM9iAMq1LTUPMiIpIncRrRndDG8d0cHK8TUJZi+RqgY112ZGatgHuAVcAf02x3lpnNM7N5q1Zlm/9ERKRCrQnE3bcCS4FtcnTMmtpS6up3wAHASe6eKimFg7lPdPdB7j6oS5cuGRxGRERSiduN917gYjNrU+uW6ZWRustvR1LXTFIysxuBs4DT3f35LGMSEZEMxG1E3w74FvCxmT0LrKBqTcLd/eoY+ykhtIMk6wcsiBOImV0FXAFc6O4PxSkjIiK5V2MCMbOPgaPcfT7wy4RVp6fY3IE4CWQ6cIuZ7eLuH0fHKQIGE5JCWmZ2IeG+kas0iZWISMNKdwmriHCXOe7eopZH3JF4/0C4p2SamY0ys5GEXllLCZfJADCzPma22czGJSz7KXAH8Czwkpntn/DoF/+URUQkF+o0oVS23P0rMxsG3E6YHteAF4GL3f3LhE0NaEnVBHdYtPyw6JFoFlCcp7BFRCSF2hJI2jlAMuHuS4Cja9lmMUk9s9z9VODUXMcjIiKZqS2BXGtmq2Psx919dC4CEhGRpqG2BLI3YeDD2uS8piIiIo1bbQnkSHd/o14iERGRJiUX84GIiEgzpAQiIiIZUQIREZGM1NgG4u5KLpJT69atY+XKlWzatKmhQxFp9lq3bk3Xrl1p3759xvuo1xsJpflat24dn332GT169KBdu3aYZTIAs4jkgrtTXl7OsmXLADJOIqplSL1YuXIlPXr0oLCwUMlDpIGZGYWFhfTo0YOVK1dmvB8lEKkXmzZtol27dg0dhogkaNeuXVaXlJVApN6o5iHSuGT7P6kEIiIiGVECERGRjCiBiGTgwQcfxMwqH9tttx177bUXv/vd79i8eTMAM2fOxMyYOXNm3uI49dRTKSoqqny9ePFizIwHH3wwb8dM59Zbb2XPPffEvfrweK+99hpmRrdu3Srfo0QV79eMGTNS7ruoqIiTTjqp2vLZs2dz3HHHsdNOO9GmTRs6d+7MD37wAyZNmsSWLVuyP6mYSkpKOPTQQ9l2223p3Lkzp512GmvWrIlV9u233+awww5j2223pX379owcOZKFCxdW2660tJRjjjmGDh06sM0223DwwQczb968KtusWLGCwsJC3ngj/6NQKYGIZOHxxx9n9uzZTJkyhX333ZcLLriA8ePHAzBw4EBmz57NwIED6y2e7t27M3v2bA4//PB6O2aFL774ghtuuIFx48alvLY+adIkIPTI+/vf/56TY95xxx0MHjyYNWvWcPPNNzNjxgzuv/9+vv3tb3Puuefy9NNP5+Q4tVm+fDnFxcWUl5fzxBNPcNdddzFjxgx+/OMfs3Xr1rRlP/roIw466CDWrl3LI488wgMPPMDixYsZMmRIlR5Sn3/+OQceeCDvvvsu9957L48++igABx98MO+9917ldt27d+fMM89k7Nix+TnZRO7ebB7f+973PBPH3fMPP+6ef2RUVoIFCxY0dAg59cADDzjgH330UZXlxcXFvt1229VbHKNHj/Y+ffrU2/HSueWWW7xbt26+efPmaus2bNjg22+/vRcXF3thYaEfffTR1bZ5+eWXHfAXXngh5f779OnjJ554YuXrWbNmuZn5BRdckHL7hQsX+vz58zM8m7q5+OKLffvtt/eysrIq8QE+ZcqUtGXHjBlTrezSpUu9oKDAx44dW7lswoQJ3rJlyyp/c19++aV37drVjz322Cr7LCkpccDnzp1ba+y1/W8C87yGz1TVQERyaJ999mH9+vWsXLky5SWs4uJiDjzwQKZNm8aAAQMoKChg99135y9/+Uu1fc2fP5+RI0fSsWNH2rVrx+DBg3n11VfTHj/VJaxTTz2Vnj178q9//YuDDjqIwsJCdt11V+65555q5UtLSznxxBPp0qULBQUF7L333jz55JOxzv2+++7j+OOPp2XL6jNcP/XUU6xdu5bzzjuPo446iqeffpqysrJY+63JTTfdRKdOnfj1r3+dcv23vvUt9txzz6yOEdf06dM5/PDD6dChQ+WyIUOG0Lt3b6ZNm5a27Jw5c/j+979fpWzPnj0ZMGBAlfd+zpw57LrrrvTt27dy2TbbbMNBBx3E008/XeWyYL9+/fjOd77Dfffdl/3JpaEEIpJDpaWltGzZkm233bbGbRYuXMiFF17IZZddxtSpU+nbty8//elPefnllyu3eeuttzjggANYs2YNf/jDH5gyZQqdO3dm+PDhvPnmm3WOa926dZxwwgmcdNJJTJs2jX322Ydzzz23yjGXLl3Kfvvtx/z587n99tuZPn06AwcO5Oijj2b69Olp979kyRLef/99DjrooJTrJ02aRIcOHRg5ciSnnHIKGzdurLwEk4ktW7Ywc+ZMDj30UNq2bZvxfjZv3hzrkU55eTmlpaUMGDCg2rr+/fuzYMGCtOVbtmxJmzZtqi0vKChg0aJF/Oc//6l1u/LychYtWlRl+ZAhQ3j22WfTHjtbGspEGtS1fy1hwfJ1DRpDv53ac/UR/TMqu2XLFjZv3sz69ev5y1/+wtSpUzniiCMoLCysscxnn33G7Nmz2X///QE47LDD6N+/P+PGjausYYwdO5bevXvz0ksvVX5ojBgxggEDBjBhwgSeeuqpOsW5fv16fv/733PwwQcD4cPl+eef589//nPlsmuuuQZ3Z9asWXTu3LnymEuXLmXcuHGMHDmyxv3Pnj0bgL322qvauuXLlzNjxgzGjBlDQUEBw4cPp0ePHkyaNIlzzz23TudRYfXq1ZSXl9OnT5+Myldo3bp1rO08RaeACmVlZbg7HTt2rLauU6dOfPDBB2n3vdtuu/GPf/yDTZs2Vcazfv16SkpKcHfKysro3r07u+22Gy+88AKff/555e9n69atlY3lyQ323/3ud7nrrrtYvnw5O+20U6zzrCvVQESysPvuu9O6dWs6derEeeedx4knnsj999+ftkyvXr0qkweEb5bHHnssb7zxBlu3bqW8vJxZs2Zx7LHH0qJFi8pvwe7O8OHDeeWVV+ocZ2FhYWWigPCtddddd2XJkiWVy5599ll+9KMfsf3221f59j1ixAjmz5/PunU1J/rly5cD0KVLl2rrHn74YbZs2cIpp5wCQIsWLTjppJOYO3durR+u+fbPf/4z1iOdiuSSquNAusRT4aKLLmLZsmWcc845LFu2jE8++YTTTjuNL7/8EgjvF8A555zD1q1bOeWUU1i0aBErVqzgwgsvpLS0tMp2FSp+FxW/m3xQDUQaVKbf/BuLJ598kp49e7LddtvRp0+fWJdTunXrlnLZ119/zapVq9i8eTNbtmxhwoQJTJgwIeU+tm7dWu0DI51U344LCgoqL49A6B01efJkJk+enHIfn3/+eY2D7lXsp6CgoNq6yZMn07t3b/r3788XX3wBwKhRo7j55puZPHky119/PQCtWoWPo5q63m7ZsqVym86dO9OuXTs++eSTlNvGtffee2dVHsJ7a2Ypu+yWlZXRqVOntOUHDx7MXXfdxZVXXln55eOQQw5h9OjRPPzww5Xld9llFx555BHOP//8ynaQgQMHcskll3DLLbfQvXv3KvutGDqovLw863OsiRKISBYGDBhQpVEzjs8++yzlsjZt2tClSxfKy8tp0aIF559/fuW39mR1SR5xde7cmYMOOohf/OIXKdenuwxScUmlrKysyphn8+bNo6SkBEidxB566CEmTJhAixYt6Nq1K5D6G/PmzZtZuXJlZfJt1aoVxcXFvPDCC2zcuDFl4oojF5ewCgsLKSoqqjzPRAsWLGDo0KG17v+8885jzJgxLFy4kPbt29OrVy9++MMfst9++1WJ8eijj+bII4/kww8/pE2bNnzrW9/i3HPPpVevXvTu3bvKPisS2g477BDrHDOhBCJSz5YuXcqcOXMqL2Nt2bKFxx9/nH333ZcWLVpU9qyZP38+AwcOzEuySOWwww5j9uzZ9O/fv84DX+6+++4AfPzxx1USzaRJkzAznnjiiWrfxJ977jluuukmZs6cybBhw9h1113p2bMnU6dO5bTTTquy7dNPP83XX39d5TLcFVdcQXFxMWPHjuXOO++sFlNpaSnr169P2xOrtstTcY0cOZJJkyaxdu1att9+eyDcOPnJJ5+kbTtKVFBQQP/+oUb+f//3f8yYMSNlbbBly5bsscceQEi2jz32WMp7PkpLS2nTpg0777xzpqdVKyUQkXrWrVs3jj/+eK699lq6dOnC3XffzYcffsjdd99duc1tt93GkCFDGDFiBGPGjKF79+6sXr2at956iy1btnDTTTflPK7x48ez7777MmTIEH7+859TVFREWVkZ7777Lh9//HHatp19992XgoIC3njjDQ488EAgjMD86KOPMnToUH7yk59UK7P33ntzxx13MGnSJIYNG4aZceONN3LyySdz9NFHc8IJJ9C+fXvmzZvHDTfcwLBhwxgxYkRl+SFDhnDbbbdx6aWX8t5773HqqafSu3dvysrKePHFF7nvvvv405/+lDaBDBo0KIt37L/Gjh3Lww8/zMiRI7nyyitZu3Ytl19+Ofvuuy9HHXVU5XazZs3ikEMO4f7776+sXf773//m7rvv5oADDqCgoIA333yTG264gZ/85Cf87Gc/qyy7adMmLr/8coYOHUr79u0pKSnhxhtvpH///lx22WXVYpo7dy777LNPVr3UalXTDSLfxIduJGw4zeVGwkQVN8a9/PLLlcuGDh3qgwcP9mnTpnn//v29TZs2/u1vf9sfffTRauUXLFjgxx9/vHfp0sXbtGnjPXr08COOOMKfeeaZym2SbyQsLS11wB944IEq2/To0aPa/ocOHepDhw6tsmzp0qU+ZswY32mnnbx169a+4447+vDhw/2hhx6q9T057rjjvLi4uPL11KlTHfDJkyfXWOaEE07wbbbZxtevX1+5bPr06T5kyBDfdtttvXXr1t63b1+/8sorfcOGDSn38frrr/sxxxzjO+64o7dq1co7duzoP/jBD/yhhx7yLVu21Bp3rrzzzjs+fPhwLyws9A4dOvjo0aN99erVVbap+JtI/P18+umnfsghh3jnzp29TZs2vscee/gtt9zimzZtqlJ206ZNfvjhh3vXrl29TZs2vssuu/hVV13lX331VbVYNmzY4Nttt53/9re/rTXubG4kNI/RS+CbYtCgQZ48bkwcx98buig+dvb3cx1Ss/Hee+9VVrubs+LiYjZv3sxrr73W0KHkXMWlqMWLF1e7Hi/167HHHuOMM85g6dKlVW5QTKW2/00ze9PdU1bV1I1XRHKiuLiY4cOH13hnuNSfm2++mbFjx9aaPLJV7wnEzHqZ2RNmttbM1pnZVDOL9XXFzNqa2W/MbIWZlZvZbDMbku+YRSSeO++8k549e8a6/0Hy49NPP2XUqFH8z//8T96PVa+N6GZWCLwEbARGAw5cB7xsZnu6+1e17OKPwOHAWOBj4HzgOTP7vru/nbfARXIkn0O7Nwa77747V1xxRUOH0aztuOOOXH311fVyrPruhXUmsAuwm7svBDCzd4CPgLOB22oqaGZ7AScAp7v7A9GyWUAJMB6I11dORERyor4vYY0E5lQkDwB3LwVeB0bFKLsJeCyh7GbgUWCEmWV2J5GIiGSkvhNIf+DdFMtLgH4xypa6+4YUZdsAdbsdWOqdrouLNC7Z/k/W9yWsTkCqSQDWANXHOYhftmJ9NWZ2FnAWkHHXwn47pR7/R+Jr3bo15eXlaUepFZH6VV5eHns4l1Qa4k70VCmv+jCWqbepc1l3nwhMhHAfSIzjVNPUB/xrDLp27cqyZcvo0aMH7dq1SzlyqYjUD3envLycZcuWpRzcM676TiBlpK4pdCR17SLRGiBVFaJjwnpppCpGcV2+fDmbNm1q4GhEpHXr1nTr1q3GEZbjqO8EUkJoy0jWD0g/bVcoe5SZFSa1g/QDvgYWpi4mjUX79u2z+mMVkcalvhvRpwP7m9kuFQvMrAgYHK2rrWxr4NiEsq2A44Hn3X1jzqMVEZEa1XcC+QOwGJhmZqPMbCQwDVgK3FuxkZn1MbPNZjauYll0o+BjwB1mdoaZHULowrszUD93zYiISKV6TSDRnebDgA+Bh4BHgFJgmLt/mbCpAS1TxHca8ADh7vVngF7AYe7+Vp5DFxGRJPXeC8vdlwBH17LNYlL0rnL3cuDS6CEiIg1Io/GKiEhGlEBERCQjzWpCKTNbBXySYfEdgNU5DKcp0Dl/8zW38wWdc131cfcuqVY0qwSSDTObV9OsXN9UOudvvuZ2vqBzziVdwhIRkYwogYiISEaUQOKb2NABNACd8zdfcztf0DnnjNpAREQkI6qBiIhIRpRAREQkI806gZhZLzN7wszWmtk6M5tqZrGmLTSztmb2GzNbYWblZjbbzIbkO+ZsZXrOZjbIzCaa2ftmtsHMlpjZI2a2c33EnY1sfs9J+7nSzNzMXstHnLmU7Tmb2R5m9riZrY7+vj8ws4vyGXO2svx/7m1mk6K/6w1m9qGZXWdm2+Q77kyZWU8z+2302bMh+tssilk2J59fzTaBmFkh8BKwOzAaOBnYFXg55h/NH4EzgXHAj4EVwHNmtndeAs6BLM/5p4S5XO4EfghcAQwE5plZr7wFnaUc/J4r9rMLcBWwMh9x5lK252xmg4C5QAFwBvAj4FbCAKeNUjbnHK2fAQwBfgUcDtwHXAbcn8ews9UXOI4wGd+rdSybm88vd2+WD+AiYAvQN2HZzsBm4NJayu5FmF73tIRlrYAPgOkNfW55OucuKZb1AbYC4xv63PJxzkn7eY4w5cBM4LWGPq88/p5bECZve7Khz6Mez/nQ6P/50KTlN0XlCxv6/Gr6XSX8fEZ0DkUxyuXs86vZ1kCAkcAcd6+cydDdS4HXgVExym4izE9SUXYzYX6SEWZWkPtwcyLjc3b3VSmWfQKsAnrkOM5cyub3DICZnUCobV2ZlwhzL5tzLibM8nlb3qLLj2zOuU30vC5p+ReEhFptZPDGwN23Zlg0Z59fzTmB9AfeTbG8hPAPVFvZUq86tW5F2TaEqmVjlM05V2NmewBdgfeyjCufsjpnM+sI3A5c7u5rchxbvmRzzgdGz23NbI6ZbTKzlWZ2p5m1y2mUuZXNOc8APgJuNrN+ZratmQ0j1Gru8TCP0TdJzj6/mnMC6US4dphsDdAxi7IV6xujbM65img64XsINZA/Zh9a3mR7zr8hTID2YA5jyrdsznmn6Pkx4HngB8CvCZdI/pSrAPMg43N29/8QEmfF5bv1wIvA08DPcxtmo5Czz696n1CqkUl1F2Wc6qplUbah5Sru3wEHAIe7e6o/xsYko3M2s4OAU4CBHl0obkIy/T1XfKl82N0rppSeaWYtgZvMrJ+7L8hJhLmX6e+5LSFhdiU0vi8B9iU0MG8Gzs1hjI1Bzj6/mnMCKSN1pu1I6uycaA2Qqntgx4T1jVE251zJzG4EzgJGu/vzOYotX7I553sJtat/m1mHaFkroGX0utzdN+YozlzK5pw/j55fSFr+PKFReW+gMSaQbM55DKHtp6+7L4qWvWJma4GJZnaPu8/PWaQNL2efX835ElYJ4Vpgsn7U/g9SAuwcdR1MLvs1sLB6kUYhm3MGwMyuInThvcjdH8phbPmSzTnvAZxD+ACqeAwG9o9+bqzfTLP924bq31Arvp1m2nCbb9mc83eAsoTkUeGN6HmPLGNrbHL2+dWcE8h0YP+ofz8A0U04g6N1tZVtDRybULYVcDzwfCP9VgrZnTNmdiFwHXCVu/82X0HmWDbnfHCKx3xCY+3BwBN5iDcXsjnnvwMbgcOSlo+InuflKMZcy+acPwU6mlly4/F+0fOyXAXZSOTu86uh+zI3YB/qbQiZ9v8I3fxGEj4cPga2TdiuD+E66Lik8o8SvoWeARxC+DD5D+F6eYOfX67PmXAj4VbCB8z+SY9+DX1u+fo9p9jfTBr/fSDZ/m1fHS2/ARhOqHGWAw829Lnl45yBIkIX3g8JNyEeDIyNls0j4X6LxvYAjokedxNqjedGr4fW8jvOyedXg78BDfzm9wamRH8o64GnSLoRJ/rjcuCapOXtCH3lP43e+LlAcUOfU77OmdALyWt4zGzo88rX7znFvhp9Asn2nAmXqy6NPpC/JkwDPR5o3dDnlcdz7gf8BVhKSJYfArcAHRv6vGo557T/k/n+/NJw7iIikpHm3AYiIiJZUAIREZGMKIGIiEhGlEBERCQjSiAiIpIRJRAREcmIEojknZmdGk23meoxvA77WWxmD+Yx1OTjJca52cxKzewBM+uZ4+MURcc4NWHZqWZ2eoptK97LolzGUEt8xSneiyVm9vtouPtM9nmxmf0k17FK/WrOgylK/TsW+HfSssY4MF+iBwmDKrYiDCR4LXCAme3t7uU5OsYK4PtA4lhMp0bHTJ5S9Zlo2xU5OnZdXAj8Eygk3L38C6AXcEQG+7oYeA2YmqvgpP4pgUh9etsTZoxrIpa5+5zo59fMbD0hqfyQHH34eRh7aE6tG1I5M2S12SHryXsJ78VLZtYVOMPMdnT3TxsoJmlAuoQlDc7MDjWzv5nZCjPbYGbvmtll0RwU6crtaGaTzGy5mW2Myj8dfbBVbFNoZjdHl5++jp6vMrNM//b/GT33jfbf3cwmm9nqKIZ3zOykusSZfAnLzGYCQ4HBCZeNZkbrqlzCit63N1O8N92jS00XJyzb2cweMbNVURxvm9lRGb4PAG9Fz5VDg5vZPmb2hJn928zKzewDM7vBEmYzNLPFhDGaTkw4vwcT1u9lZtPNrCzax+sW5maRRkY1EKlPLaNRPyu4u28BdiHMAPdbwrg8g4BrgC6Egfxq8hDhg2gsYQyjboRLK4VQOcLoc4RxjiYQBtrbH/gVYe6IyzI4h52j5y/MbBtgFmEehV9GMZwEPGRmhe4+MU6cKZwHPAy0BM6OliXP111hMvBnqz7R0wnR858BzKwXYbyjlcAlhFrM8cAUMzvS3WsdjTmFImALsDhhWW/gbUItbT1hiPVxhN/xT6NtjgL+Rhjs8Jpo2aoozoHAq8C/gDOBDYQh9WeY2QHuXi1ZSgNq6MHA9PjmPwjX81MN+FZtUELCQH6tgKsIo4W2SFi3mIQRYYEvgQvTHPfk6DhDkpZfRRgksGstcTtwfRRPW0LyeQ/4ijD168+jbYqTys0gfFC3jBlnUbSfUxOWzazh/al4L4ui1+2AtcCNSdu9Dfwt4fUfCR/SnZO2e4FwaTHd+1AcHfPQ6L3YDjiSkNRuSVOu4nd5EmEk584J6xYTZj1MLvNi9B63SVjWMlr2VEP/LetR9aFLWFKfjgL2SXiMgcrLLfea2SeED/ZNhHlHOhCmGa3JP4GxZnaRmX3HzJKn5DyMMJLsP8ysVcWDMLtea0JCqM0vo3jKgdnRzz9y9+XAEEIbycykMg8Tak/9YsaZMQ8N+VMIl4MMwMy+A+xFqJ1UOIzwrX9t0nvxHLCXmbWPcbjnCOe/DngSeIVQq6pkZu2jS4aLCPOKbCLUwAzYNd3Oo8tcQ4HHga0JMRohKQ+JEaPUIyUQqU/vuvu8hMcHUVvEdODHhKQxjJBcro/KtE2zv+OjspcD7wDLzGxcQvtGV8Klo01Jj4qZ5jrHiPn+KJ7vAju4+57uPita14nUvaE+TVgfJ85sTSb0hiqOXp9MuHw0LWGbroT53ZPfi99E6+O8F+cT3ovhhDnEDydcDkz0AOGS053AD6Ltz4/WpftdQni/Wkb7TI7z54RJn/SZ1YioDUQa2rcIbR4nu/vDFQvNrNauoe6+kvDhdL6Z7UaYDOhawqWauwnze5cCx9Wwi8Ux4lvh7jXNwrcG2C3F8h2j589jxpmtWcAS4CQzmwX8DHjCq3Yz/pzQtnBzDftYHuM4H1a8F2b2EqEt55dm9oC7LzWztoTJnK5x9/+tKBTViOL4gnCp6y6q1p4quXtjnVK3WVICkYZW0ZC8qWKBmbUGTqzLTtz9A8KH2TnAgGjxs8DRwJfu/n4OYk02CzjWzAa7++sJy08gtIG8FzPOVDYS2hpq5e5uZo8QktSTQE+qfwA/S7h/pMRzcP9KdMyLCY3dV0THLiDUIDYlbX5qil1sJLTfJO7zKzN7lXD57S0li8ZPCUQa2nuEdorrzWwL4cPnktoKmdn2hOvijwDvR+VGEXpEPR9t9ghwGvCimd1K6PXThlDrGQkc6e4bsoj9QeAiYKqZXUW4SfJEwqWbs919S8w4U1kAnGdmxxNuMFwfJZ+aTAauBO4h9PSalbR+HOHS3Stm9jtC7asjIYnt4u7V7nqvjbvPN7MpwBgzu97dl5vZHOAyM1sBrAZOB3rUcH4HmdmPCZf8Vrv7YsJMiK8Az5nZHwmXCHcABhI6JaTrlSf1raFb8fX45j/4b8+hvjWs35twV/IGwofweMJczZW9jaLtFhP1wiJ8270XKCH0clpHaKw+IWnfbQldRd8nfOtdE213DdCqlrgduK6WbboTGolXR/t/BzgpYX2tcZK6F9aOhEbv9VSdorTivSxKEcs/o3U31BBrT+A+YBmhs8IKQi+sk2o5x+Jov8NTrNuD0JX3fxPO5e9R3CuB3xHaSqr0VgN2J1xS2xCtezBpn49G5TdGfxPTCZ0XGvzvWY//PjSlrYiIZEQ9GkREJCNKICIikhElEBERyYgSiIiIZEQJREREMqIEIiIiGVECERGRjCiBiIhIRv4/XevzMluX1qEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(pipe_lr, X_valid_fold, y_valid_fold);\n",
    "plt.plot(cm[0,1]/(cm[0].sum()), cm[1,1]/(cm[1].sum()), '*r', markersize=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Unlike the precision-recall curve, here the top-left is the best.\n",
    "- Unlike the precision-recall curve, this curve is monotonic.\n",
    "- The red star corresponds to the threshold of 0.5, which is used by `predict`.\n",
    "  - The default threshold of 0.5 doesn't look very good here (more on this later).\n",
    "- Note: there are functions `precision_recall_curve` and `roc_curve` (without the `plot_`) if you want to raw numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(optional) Some extra explanation \n",
    "\n",
    "- If the threshold is 1, we always predict \"negative\".\n",
    "  - the true positive rate is 0 because there are no true positives\n",
    "  - the false positive rate is 0 because there are no false positives\n",
    "  - this is the bottom-left point in the curve\n",
    "- If the threshold is 0, we always predict \"positive\".\n",
    "  - the true positive rate (recall) is 1 because we have identified all positive examples\n",
    "  - the false positive rate is 1 because we have falsely identified all negatives as positive\n",
    "  - this is the upper-right point in the curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A popular metric is the area under the curve (AUC):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This animation shows what happens when you sweep the threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://raw.githubusercontent.com/dariyasydykova/open_projects/master/ROC_animation/animations/cutoff.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Area under the curve (AUC)\n",
    "\n",
    "- AUC provides a single meaningful number for systems performance \n",
    "- AUC of 1.0 means perfect classification and AUC of 0.5 means random chance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9885166695730907"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://raw.githubusercontent.com/dariyasydykova/open_projects/master/ROC_animation/animations/ROC.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Article claiming that precision-recall curves are better than ROC curves for imbalanced datasets: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Addressing class imbalance (20 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Unbalanced training sets\n",
    "\n",
    "- This typically refers to having many more examples of one class than another in one's training set.\n",
    "- Our Credit Card Fraud dataset is unbalanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    241672\n",
       "1       413\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lot of problems are coming from this class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Do I want to address the problem?\n",
    "\n",
    "- An important question is whether the class imbalance is a problem.\n",
    "- Ask yourself, do I care more about false positives vs. false negatives?\n",
    "- If not, then maybe it's fine to have a confusion matrix like we had:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2XElEQVR4nO3dd5xcVf3/8dc7uymkJ5AEEggJBFAwgF9RiD8ERCUoXbpKlSIlIiJNWihKEQUVQhEFASkCiqBSjBQLCQpICxBaQggkpPe++/n9ce4mk9nZ3bthM7vZfT8fj/uYnXs/c++Z2WQ+e8o9RxGBmZlZS9KuuQtgZmZWzMnJzMxaHCcnMzNrcZyczMysxXFyMjOzFqeyuQtgH98GvSti0Cbtm7sY1ghvvty5uYtgjTSf2TMios+avn74F7vEzFlVuWKff3npYxGx55peqzVwcmoFBm3Snv88tklzF8MaYXj/7Zu7CNZIo+P+9z7O62fOquI/jw3MFVux0VsbfJxrtQZOTmZmZRBANdXNXYx1hpOTmVkZBMHyyNesZ05OZmZl45pTfk5OZmZlEARVni4uNycnM7MyqcbJKS8nJzOzMgigyskpNycnM7Mycc0pPycnM7MyCGC5+5xyc3IyMyuDINys1whOTmZm5RBQ5dyUm5OTmVkZpBkiLC8nJzOzshBVqLkLsc5wcjIzK4M0IMLJKS8nJzOzMkj3OTk55eXkZGZWJtWuOeXm5GRmVgauOTWOk5OZWRkEoop2zV2MdYaTk5lZmbhZLz8nJzOzMgjEsqho7mKsM5yczMzKIN2E62a9vJyczMzKxAMi8nNyMjMrgwhRFa455eXkZGZWJtWuOeXm5GRmVgZpQIS/cvPyJ2VmVgYeENE4Tk5mZmVS5fuccnNyMjMrA88Q0ThOTmZmZVLt0Xq5OTmZmZVBmvjVySkvJyczszIIxHJPX5Sbk5OZWRlE4JtwG8GflJlZWYjqnNsanV16VFJIuqxofy9Jt0iaIWmhpNGShpZ4fSdJP5E0RdJiSWMk7VIirp2kcyVNlLRE0kuSDqyjTMdLekPSUknjJX0n7/txcjIzK4Mg1ZzybI0l6XBguxL7BTwE7AmMAA4E2gNPStq4KPzXwPHAhcDewBTgMUnbF8VdCowErgO+CowF7pP0taJrHw/cBDyQXf8+YJSkk/K8JzfrmZmVydoYECGpJ3ANcDpwV9HhfYGdgd0j4sksfgwwATgL+G62bzvgG8CxEXFrtu9pYBxwSXYeJPUFfgBcERFXZ9d4UtIQ4Argr1lcJfAj4I6IOK8grj9wqaRbImJ5fe/LNSczszIIRHXk2xrpKmBcRNxd4ti+wIc1iQkgIuYCDwP7FcUtB+4tiFsB3AMMl9Qx2z0c6ADcWXSdO4GhkgZnz4cBfUrE3QGsT0qY9XJyMjMrgwCWR2WuLS9JOwNHAifXEbIN8GqJ/eOAgZK6FsRNiIhFJeI6AEMK4pYCb5eIA9i6II4S1y6Oq5Ob9czMykKNWc9pA0nPFTy/OSJuXu1sUntSn87VETG+jvP0BiaW2D8re+wFLMjiZtcT17vgcU5ERI44SpyzOK5OTk5mZmUQNGqGiBkRsUMDMWcD65H6duqi7NKl9q/tOOqIzcXJycysTJpqJVxJA4HzgOOAjgV9QmTPewLzSTWVUrWUXtljTc1mFjCwnrhZBY+9JKmo9lQqjuzaUwriehcdr5P7nMzMyiBCVEe7XFsOmwGdSAMOZhdskEbTzQaGkvp4tinx+q2BSRGxIHs+DhgsqXOJuGWs6mMaB3QENi8RB/BaQRwlrl0cVycnJzOzMkgDIipybTm8CHyxxAYpYX2RlFAeAgZI2rXmhZK6A/tkx2o8RLr/6eCCuErgUODxiFia7X6UlKy+WVSebwGvRsSE7PkYYEYdcbOAfzf0Bt2sZ2ZWFmqy6YsiYg7wVK0rSADvRcRT2fOHSIniTklnkmpU55L6hK4qON+Lku4Frs0GWkwATgIGU5BgImKapGuAcyXNB14gJbDdKRiaHhHLJV1Auun2A2B0FnMsMCIiljX0Hp2czMzKIA2IKO9igxFRLWlv4GpgFKkpcAzwxYh4vyj8GNLgisuAnsBLwJ4R8UJR3HmkEX6nARsC44FDIuLhomvfKCmAM4AzgUnAqRExKk/ZnZzMzMpkbS+ZEVE7+0XELFKN5dgGXrsY+H621RdXRUpgl9UXl8XeRBru3mhOTmZmZVAzQ4Tl4+RkZlYm1R6DlpuTkzWpH35jM55/qjuHnzaVo8+eunL/xPGduP2qDXn9hS4snNeOfpssY/hhszjguOlUZP8KP5rcnlHnb8y749ZjzsxKOq1XzaBPLOGQUz7is7vPX3muye905OHbNuClZ7oy5b0OdO5azZbbLeLIs6aw+TZLVivP1d/bhDde6MLMqe2proaNBi3jq4fPZO+jZ1Dhdd8abYONlnHIKdPYYtvFbLbNYjqtFxz5uU/y0eQOq8V17bGC4y6Ywuf3nEvHTsHrz3fmxov6M/GN9Zqp5M0vApZXOznl5eRkTebJP/Zkwmu1v3xmTq3krAOHsP6Gy/nOxR/Qo/cK/vevrtxyaX/mzKjkuPPTPXqLF1bQo/cKjjp7ChtstJxF89vxyF3rc/63NueCWyaw89fmAvD809146ZmufPngWQwZupiF8yq4b1RfTtt7S67501tsse3ilddetqQd+x07nY0GLUPAc09344YLB/DhxI6cdOkHZflcWpP+g5axyz5zeevl9Xj12S7ssNuCElHBxbdNpN8myxh1/gAWzKng0BHTuOr+dzj5K1syY0qHEq9p/VKznpNTXmVNTpKOBm4F5gKDI2J2wbFK0qy4F0fEyDKXaxhwLfApoDPw6Yh4sZxlKCUb6VL2z2NNLJhbwU0jB3DiyA+44pRBqx17dnQP5s6q5Gd/eouNN0+3S2y/8wKmTOzI3+/vvTI5DdpqCd//2eoDiHb88jyO3HFrHr+n98rktNt+s9n3mBmooPl++/83nyN33Jo/3tKHs34xaeX+H9743mrn+8xu85k1tT2P3dPbyWkNvDK2C4dtl+6r3PMbM0smp2HD5/GpHRdy1kGb89IzaV7R157vwu1jX+fgk6dzwwUDylrmlqSpZohoC5orjfcgzQvVUvyalKj3IU31/mbzFmfdc8tlG7HpVkv44gFzah1bviz9h+zcrWq1/V16VFFdXf95KyqhS/cqKtuvmimlx/pVqyUmgC7dqxmw2VJmTm3fYFm79VpBReUaT/nVppUYDFbLTnvMY8aUypWJCWDR/ArG/q07w4bPXZvFa9FqhpKvhSUzWqXmSk6PAyMkbdhM119JUjtgK+AvEfFERIwtMW18TWzHUvvbulef7cLo+3tz6o8nlzy+yz5z6NF7BdeftzFTJ3Vg4fx2/PuRHvz9/l4ceOL0WvHV1VC1AmZNq+R31/Tjg3c7ss/RM+otw7zZFUx8oxObDFlS61hEOt+CuRX88y89GH1fb75+Qu3rWtPYdMslTBzfqdb+997sRL+Nl9Opc1WJV7UFTTp9UavXXJ9Czfj48+qNAiR9LlvzfoGkhZL+LulzRTG3SZos6dOS/ilpkaS3GlqvPmtmrCJ9DhdICkkTi845TNIzkhaT3VEt6TBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUml5sNqcVYsh5+fvTEHfWcamwxZWjKmV58VXPPwm0x6qyNH7bQ1X99qWy49bhCHnDKNQ06ZViv+lkv787WB23P49p/ivlF9OfeG9/j0F0r1bawy6vwBEOKA42snnWdHd+drA7fnwE8O5UcnDGLfY2fwzdM/WrM3bA3q1nMFC+bW7jGYP6ciO95WkxNUo1ybNd+AiCmk9ee/J+nqiHivVJCkbYGnSZMEHk2qGZ8DPC1pp4h4qSC8O2mJ4mtJywofA9wgaXzhKpBF/kJakfFfpKa9W0gLadXoQVoJ8mrgh0BNT/tmwP2kZYmrgV2AWyStFxE35vwMCo3Mzv8zUq1yB1af96rF+v31/Vi2pB2Hf7fuL/s5Myu49LjBdOpczfm/mkD3XlW8+K+u3P3zfrTvEBx66uoJ6oDjp7Pb/rOZNa09o+/vxRWnbMr5N09kp6/MK3n+e37Zlyf/2Jvv/3QSAwbXnhVl6I4L+OUj41k4r4L//asbD9zYByk45pypJc5mH5eUaqu19pe/KC1KGq3nIaJ5NedovSuBE4GLqPvO5QtJyeJL2VxSSPobafGsi4CvF8R2A06uSUSS/gHsARwOlExOETFdUs2gjMkRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEWkuqkYlJ0m9gNNJi4n9INv9uKQqUvKr63UnACcADBzQPL/GaZPbc/cv+nH61e+zfFk7lhfkheVL27FgbgXrda3ivuv78dH7Hbj9P6+t/Kt5u88voLoabv/Jhux5+Ex6rL/qr+k+/ZfTp/9yYDE7fWUeZx44hF9d0r9kcvrz7etz6+X9OersKQw/vPQs/F26V7Pldunvik9/YQHt2wd3XduPfY6ayQYbLW+6D8QAmD+nkm49V9Ta3zX73dfUoNoa34TbOM3WuJlNqfFT4EhJW9URtgvw55rElL1uHqlWsWtR7KLCGlI2i+5blF6jJK8VwJ+Ld0raQtLd2YSGy7PtOFLfVWMNBboAvy/af099L4qImyNih4jYoc/6zfOffcqkjixb0o4rT92UAz85dOUGcP+NfTnwk0OZ+Pp6THijE/0HLa3VnLPV9otYsbwdH06svytvi20XlYwZfX8vrjt3Yw48cRrfOC1/M90W2y2iulpMndQ2hzSvbe+92ZFNt6zdxDtwiyV8NLk9Sxa1zeQEbtZrjOa+z+kaYASpGa54anWovVBVjamsWtyqRqklhpeSJjpcU9OyeaRWktQV+BuwiNTE+A5pCvmTaGDuqjpslD0Wf7u2+E6RzbdZzFX3v11r/1kHDeFLB85i+OGz6D94Kb37ruD157owf07Fagnqjf91AWD9DeuuvVRXw7j/dqH/pqt/2f37kR789PSB7PmNmZxw0YeNKvcrY7oiBRttWrqPzD6eMY/3YPhhsxm60wJeGZtG7HXuWsVOX5nHkw/2bN7CNaPmmPh1XdasySkiFki6nFSD+kmJkFmkWW+LbUiOlRSbQKnxxsOATYEvRMS/anZm92kVqhk2Vvzn+fpFz2uSbz9WLdBV87xF69qjiu0+X3qgQt+Nl608ttcRM3jiD7344eGbc9BJ0+jeawUvj+nKAzf24f99dQ59B6TkdMfVGzJ/TgVbf3YhvfsuZ9a09jx2d2/G/68z51y/qlvylbFduPzkTdnsk4v5yiGzeP35Veujte8QDBmamvCeHd2dx+/tzU5fmUufActZvKAd/32yO4/cuT5fO2Im629Yu+nJGrbzXnMA2CL7nD+7+zzmzKxk7sxKXhnblbGPdee15zpz9nWT+NWlG628CVeC+0b1bcaSNz+PxMuvuWtOkKZx/z6lZ7h9GthLUreImA8gqRvpfqSnylbC1dV8E678cz/rN9qvKO4jUs3tU0X79yp6/jKwEDgEeKJg/2Efu6QtxCc/s4if/vEtfnfNhtx44QAWzk/TF33z9I848MRVgyGGDF3EH2/pw1N/6smi+RX06rOCzbZezE//+DbbfG7hyrgX/9WN5Uvb8farnfn+fluudq1+Gy/j9v+kRTY32nQpUQ23XbkRc2dW0qV7FQMGL+XMX0xit/1LVbQtjwt+tfr4pRFXpJuZX3qmC2cdNIQIceGRgzn+wg8Z8eMPaN+pmtef68JZB2/O9A/bblNqhFjh5JRbsyeniFgq6RLg5hKHLwX2Bv4u6UpSTeZsUoK4pHylXM0zwDzgekkXkfqLziet+tijJigiIlu869uS3iStebIXsFvhySJiTrZ413nZ4l2PA58Fvl2G97JWPPbhi7X2ffIzi7jsznfrfd2w4fMYNrz0iLxCR/xgKkf8oOGRdgO3WMqFv57YYJw1zvD+2zUYM39OJT/7/kB+VobyrEvcrJdfS0njt5IGL6wmIl4mfZnPA34L3EFa5GrXomHkZRMR04EDgArScPLLSUPQ7ywRfhrwB9JQ8XtJ/V8jSsSNBH4MHEEa7LEHqXZoZq2EZ4honLLWnCLiNuC2EvtXAFsW78+OPQt8uYHzHl3H/t1ylGkFJW7BqOuc2bEngE+XODSyKG4OKeEUU1FcFan2dX59cWa2bnPiya/Zm/XMzNoC3+fUOE5OZmZl4nuY8nNyMjMrgwhY4cUGc3NyMjMrEzfr5efkZGZWBu5zahwnJzOzMsmzWKMlTk5mZmXiARH5OTmZmZVBhPucGsPJycysLESVR+vl5uRkZlYm7nPKr87kJKma0ktGlBIR4URnZlYHr+fUOPUllEvIn5zMzKw+kfqdLJ86k1NEjCxjOczMWj2P1suv0U1x2TLl6wMfRkTd62ubmdlK4QERjZL7k5K0t6QXgLnAu8DQbP8tkr6xlspnZtZqROTbLGdykrQ/8CfSaq9ns/o6QxOAo5q8ZGZmrUyEcm0NkTRc0hOSpkpaKmmypN9L2roorldWgZghaaGk0ZKGljhfJ0k/kTRF0mJJYyTtUiKunaRzJU2UtETSS5IOrKOMx0t6IyvfeEnfacRHlbvmdBFwa0TsAVxbdOxV4FONuaiZWVuTakVNk5yA3sDzwKmklbPPBbYBxkraFECSSCtr70lagftAoD3wpKSNi873a+B44EJgb2AK8Jik7YviLiUtqnod8FVgLHCfpK8VBkk6HrgJeCC7/n3AKEkn5XlzkL/P6ZPAWdnPxZXO2aQ+KDMzq0dTDSWPiLuBuwv3SfoP8AZwEPBTYF9gZ2D3iHgyixlDau06C/hutm874BvAsRFxa7bvaWAcadT2vtm+vsAPgCsi4urssk9KGgJcAfw1i6sEfgTcERHnFcT1By6VdEue8Qp5a07zgA3qODYImJ7zPGZmbdZa7nOamT3WfPHvSxq49uSq68dc4GFgv4LX7Zu95t6CuBXAPcBwSR2z3cOBDsCdRde9ExgqaXD2fBjQp0TcHaSKzM553kze5PQ34FxJPQv2RVboU4FHcp7HzKxNCkR1dbtcW16SKiR1kLQFqRltKimpQGrme7XEy8YBA7OR1zVxEyJiUYm4DsCQgrilwNsl4gC2LoijxLWL4+qVt1nvPOA/wHhS1S2Ac4BtgR7A/jnPY2bWZjWiUrSBpOcKnt8cETeXiHsW+Ez289ukJrxp2fPewMQSr5mVPfYCFmRxs+uJ613wOCeiVt2uVBwlzlkcV69cySkiJkr6P+BiUtWuCtgFeBS4MCI+zHMeM7M2Kxo1t96MiNghR9wRQHdgM1J/0N8k7RwRE0mjqkvlw+JCrI046ojNLfdNuBExGfj2x7mYmVmb1sT3MEXE69mPz0p6hFRTOgf4DqmmUqqW0it7rKnZzAIG1hM3q+CxlyQV1Z5KxZFde0pBXO+i4/Vq9O3KkvpL+mw28sLMzHJqwqHkJc4dc0hNezV9RONY1f9TaGtgUkQsKIgbLKlzibhlrOpjGgd0BDYvEQfwWkEcJa5dHFevxswQcaSkCcD7pLHt70uaIOlbec9hZtZWBVBdrVzbmpDUD/gE8E626yFggKRdC2K6A/tkxyiIaw8cXBBXCRwKPB4RS7Pdj5KS1TeLLv0t4NWImJA9H0OasKFU3Czg33neT65mPUmnAr8ARpNuwvoI6AccDvxWUo+IuD7PuczM2qQAmug+J0l/BF4AXibd6rMlcDqwgnSPE6SkMwa4U9KZpGa8c0l9QletLFbEi5LuBa6V1J50H9RJwGAKEkxETJN0DWnk9vzs+ocCu1MwND0ilku6gHTT7QekvLE7cCwwIiKW5XmPefuczgBui4hji/b/RtJtpI44Jyczs3o04bx5Y4FDSN/NHUgtWk8Bl2eDIYiIakl7A1cDo4BOpGT1xYh4v+h8x5BunL0M6Am8BOwZES8UxZ1HGuF3GrAhaQT3IRHxcGFQRNwoKbLynQlMAk6NiFF532De5LQhq8bOF7uL9CGZmVl9mig5RcSVwJU54maRaizFFYviuMXA97OtvrgqUgK7LMe1byLde7VG8ianV6jdCVZjC0rf6GVmZiut+WCHtihvcjoNuEfSDOAPEVElqYI0keCZwGFrq4BmZq2Gl8PIrc7kJOl9Vv8oe5Ca9qokzSaNba8gtT/eC2y6FstpZrZuC4g1HInXFtVXc/o7zvNmZk3IySmvOpNTRBxdxnKYmbV+/nM/t9zTF5mZ2cfk5JRbo5JTtijVVqTx8quJiNubqlBmZq1OE96E2xbknSGiJ/AXYKeaXdlj4d8BTk5mZvVowptwW728c+v9mLSC4S6kxHQAaTqK3wHvAp9bK6UzM2tNqpVvs9zJaTgpQY3Nnk+OiKci4kjSvEmnrY3CmZm1Jop8m+VPThsB72ZTVywBuhUc+wOwV1MXzMysVYlGbJY7OU0lTQYI8B4wrODYkFrRZmZWRGlARJ7Nco/W+xcpIf0ZuAO4SNIg0vTsR7H62iBmZlaKa0W55U1OFwM1K9/+hDQ44lCgMykxjWj6opmZtTLVzV2AdUeu5BQR75CtrhgRy0lrdJyxFstlZta6+D6nRvEMEWZmZeKRePnVNyv5hY04T0TEpU1QHjOz1svJKbf6ak4jG3GeAJyczMysSdQ3K3neYebWzN58uTPD+2/f3MUwswa4WS8/9zmZmZVD4KmJGsHJycysXFxzys3JycysTNysl5+Tk5lZuTg55ebkZGZWLk5OuTk5mZmVgZfDaJzGLtO+LWnBwfWBmyJiqqQhwEcRMX9tFNDMrNXwaL3c8i7T3hG4E/g6aSXcAB4mLaVxFfAmcM5aKqOZWavgmlN+eW+0/RHwZeAIoB8pQdV4hLRSrpmZ1ceLDeaWt1nvcOD8iLhLUkXRsQnAoCYtlZlZa+M+p0bJm5zWB16v41g7oGPTFMfMrBVzcsotb7PeBFZfmr3Q54DxTVMcM7PWS9X5NsufnG4HzpH0TaBDti8kfRE4HfjN2iicmZm1TXmT01XAX4A7gFnZvn8Bo4FHI+KXa6FsZmatSxMNiJB0kKQHJL0nabGk8ZIul9StKK6XpFskzZC0UNJoSUNLnK+TpJ9ImpKdb4ykXUrEtZN0rqSJkpZIeknSgXWU8XhJb0hampXvOw2/s1VyJaeIqIqIw4BdgZ8CtwC/AHaPiG825oJmZm1SrLoRt6Ethx8AVcAPgT2BG4CTgL9JagcgScBD2fERwIFAe+BJSRsXne/XwPHAhcDewBTgMUnbF8VdSlrr7zrgq8BY4D5JXysMknQ8cBPwQHb9+4BRkk7K9e5o5E24EfFP4J+NeY2ZmWWabkDEPhExveD505JmAb8FdgOeAPYFdiZVIp4EkDSGNIbgLOC72b7tgG8Ax0bErdm+p4FxwCXZeZDUl5QUr4iIq7PrPplNxHAF8NcsrpJ0+9EdEXFeQVx/4FJJt0TE8obeoBcUNDMrlyZq1itKTDX+mz0OyB73BT6sSUzZ6+aSJlDYr+B1+wLLgXsL4lYA9wDDs0kYIN3P2oE0IUOhO4GhkgZnz4cBfUrE3UEa+b1zQ+8PciYnSdWSqurb8pzHzKytEmt9tN6u2WPNbT/bAK+WiBsHDJTUtSBuQkQsKhHXARhSELcUeLtEHMDWBXGUuHZxXL3yNutdQu18vj6wB+kep9tynsfMrG1q3E24G0h6ruD5zRFxc13BkgaQvqdHR0TN63oDE0uE1wxq6wUsyOJm1xPXu+BxTkQUv4tScZQ4Z3FcvXIlp4gYWWp/NlvEw8DcPOcxM2vT8ienGRGxQ57ArAb0J2AFcEzhoTquWDz77NqIo47Y3D5Wn1NEVAGjgO99nPOYmbUJTTy3nqROpBF5mwHDI2JyweFZlK6l9MoeZ+eMm1Xw2CsbBdhQHCXO2bvoeL2aYkBExxKFMDOzIk04lBxJ7UlDtT8HfC0iXikKGceq/p9CWwOTImJBQdxgSZ1LxC1jVR/TONL3/eYl4gBeK4ijxLWL4+qVd0DEwBLbEEn7k4YQPtfAKczMrOluwm0H/A74ErBfRIwtEfYQMEDSrgWv6w7skx0rjGsPHFwQVwkcCjweEUuz3Y+SklXxva3fAl6NiAnZ8zHAjDriZgH/bvgd5h8QMZG62xrfAU7JeR4zs7YpmnTevOtJyeRHwEJJOxUcm5w17z1EShR3SjqT1Ix3Lul7+6qVxYp4UdK9wLVZbWwC6YbewRQkmIiYJuka4FxJ84EXSAlsdwqGpkfEckkXkG66/YA0k9DuwLHAiIhYlucN5k1Ox5TYtwR4D/hv1vdkZmb1abqbcL+aPZ6XbYUuBkZGRLWkvYGrSWMDOpGS1Rcj4v2i1xxDSnSXAT2Bl4A9I+KForjzSCP8TgM2JE36fUhEPFwYFBE3SgrgDOBMYBJwakSMyvsGG0xO2Yi8F0k3c5W68cvMzHJoqvWcImJQzrhZpBrLsQ3ELQa+n231xVWREthlOa59E2kKozWSp88pSH1Kn17Ti5iZGV4JtxEarDllVcP3gS5lKI+ZWevkxNMoeYeS3wR8T1KHBiPNzKwW0bRDyVu7vAMiupHGtr8r6VHSdOqFH2FExEVNXTgzs9bEiSe/OpOTpHeBAyLiJdKaITVKdawF4ORkZlYfJ6fc6qs5DSLdDUxEeGkNM7OPy8kpt0YtNmhmZmvI/UmN0lBy8kdpZtZU/I2aW0PJ6WJJM3KcJyLiqKYokJlZa9WE0xe1eg0lp+1JKx82xH8PmJk1wM16+TWUnPaPiP+UpSRmZq2Zb8JtFA+IMDMrFyen3JyczMzKoGaGCMvHycnMrExU7eyUV53JyTfempk1Ifc5NYprTmZmZeJmvfycnMzMysXJKTcnJzOzMnHNKT8nJzOzcnFyys3JycysHMLTFzWGk5OZWRn4PqfGcXKyZvOZXedxyCnTGbjlErr2qGLuzEpef64zd/x0Qya91WllXNceKzjugil8fs+5dOwUvP58Z268qD8T31ivGUvf9mw7bAE/eeCdWvsXzG3HgZ8cWvI1373yffY6YhZ/f6AnV43YdG0XseULZ6e82nRyknQ0cGsdh78SEaPLWJxaJD0FEBG7NWc51pZuvap465X1ePi36zN3ZiV9Byzj0FOnce2f3+I7u2/FtA86AMHFt02k3ybLGHX+ABbMqeDQEdO46v53OPkrWzJjSofmfhttzvXn9+fNFzuvfF61QiXjtt5hIbt/fQ4L5/mWyRquOeXXppNTgYOByUX7XmuOgrQlTz3Yi6ce7LXavvEvdubX/xzPF/aewwM39WXY8Hl8aseFnHXQ5rz0TFcAXnu+C7ePfZ2DT57ODRcMaI6it2nvv9WJN17oUm9MRWVw2lWTufvnffnaETPLVLIWzjfhNor/pElejIixRdu84iBJHZujcG3JvNnp76UV2V/jO+0xjxlTKlcmJoBF8ysY+7fuDBs+t1nKaA07+KRptKsIHripT3MXpUVRdb7NnJzqJOloSSFpF0n3SZoDPJsd+6yk+yVNlrRY0nhJP5a0XtE5Jkq6rcS5Q9LIon2HSXpD0lJJ4yQdsPbeXcvSrl1Q2b6a/oOXctqVk5n5USVPPdgTgE23XMLE8Z1qvea9NzvRb+PldOpcVebS2tnXTeKv77/Efa++yjnXv0efActWO77Rpks5/Hsfcd25G7Niub9iCjk55edmvaRCUuFnUVj5/h1wN3AQqz6vgcCLwG3AfGAb4EJgM+Cwxl5c0peBu4C/AGcAfYCfA+2B8Y0937rm5395iy23WwzAB+924OyDN2fuzPYAdOu5go8md671mvlzKrLjVSxZVFG+wrZhC+e34/4b+/DymC4sml/BkE8t5rDvTuPah97i5D22XPk7++6Vk/n3X3usVts1smY9t+vl5eSUvFH0/N/ALdnP90fEWYUHI+KBmp8lKYufB9wu6ZSIaGwj+8VZGfaLiOrsvK8DY6kjOUk6ATgBoBO1v7zXJVeNGEjnblVstOkyDvrOdC6/513O2H8IH03ugFT6/3PpLnhbm955tTPvvLrq39orY7vyyrNd+MVf3mL/b8/gt1dtxO5fn82W2y3iuF0+0Ywlbbk8ICI/17mTA4DPFmzfLjj2x+JgSd0lXSnpHdIy9suBO0jfmVs05sKSKrJr3l+TmAAi4llgYl2vi4ibI2KHiNihPet2V9j7b3di/P+68NSDvTj7kM1Zr0sVh546DYD5cyrp1nNFrdd07VmVHXetqTm9/UpnJr/bkS23X0SnzlWcOPJDfn99X5YtbUeX7lV06V5Fu3ZQ2T7o0r2Kiso2/u0cOTdzzSnzakS8XbhD0rDsxykl4m8FvkxqynsRWAh8DrgeqN1BUr8NSM13H5U4Vmpfq7ZwXgUfTuxI/8FLAXjvzY783y4LasUN3GIJH01u7ya9FkACQvToXUXPDVZw7A+ncuwPp64W03fAXHbddy4jjx3EmEd7NE9Bm5lvwm0cJ6eGrfbPSVInYD9gZET8vGB/qbsQlwCr3YgjqXdRzAxSzatfidf3A95bgzKvs3pusJxNhizliT/0BGDM4z0Yfthshu60gFfGpj6Mzl2r2Okr83gyGzRhzWeLbRcxYLOl/OPhnsyaXsmZB25eK+bcG95j4huduPvn/UoObmkzIrzYYCM4OTVeR6CClFAKHV0i9j3gU0X79i58EhFVkv4LHCRpZEGf047AIFpxcrrw1xN4+5XOTHi9E4vmVzBgs6V8/YTpVK2AB27qC8DYx7rz2nOdOfu6Sfzq0o1W3oQrwX2j+jbzO2hbzr7uPaZO6sDbr6zHgnlpQMShI6Yxc2p7/vSbDVi+tB0vj6k9CGLZUjF7emXJY22Oc1NuTk6NFBFzJY0FzpA0hVTzORYodTfoPcBvJF0D/BnYjtJJ7CLgceBBSTeRRutdDEwtEdtqvPFCF3bZZw4HnjiNyg7B9A878PIzXbj3l/34aHKqcEaIC48czPEXfsiIH39A+07VvP5cF846eHOmf+jZIcpp4hud2G3/Oex37Aw6rlfN7Ont+fdfe3DH1Rsyb5a/SvJoymY9SRsDZwM7kL5b1gMGR8TEorhewE+A/bOYMcDpEfFKUVwn4FLgW0BPUpfF2RHxj6K4dtl1TwQ2JA3auqRwoFhB7PGkEciDSX3o10TEjXnen/9FrZnDgRtIfUyLgd8Dp5ESUKHfApuQBlicCPyTNPhitf6tiBgt6ZvASOAP2fHvZedstX5/fV9+f33DtZ/5cyr52fcH8rMylMnqdu91/bj3ulKtz/U7aset10Jp1kEBNG2z3hDgEOB50nfLHsUB2Wjih0jJYQQwGzgXeFLS9hFRODPOr4G9gDOBd4FTgMckDYuIFwviLgV+AJyXXfsw4D5Je0fEXwuufTxwE3A5MBr4EjBKkiLihobenMLj7td53dU7dtSXmrsYZq3a6Lj/+YjYYU1f363HxvF//++7uWL/8cjZDV5LUruCboDjgF9RVHOStB/wILB7RDyZ7esBTADujIjvZvu2I9WUjo2IW7N9lcA4YHxE7Jvt6wu8D1wRERcVXOfvQJ+I2LbgtR8Cj0TEUQVxvwH2BTaKiOKukdV4KLmZWZko8m15FN56Uo99gQ9rElP2urnAw6SBXYVxy4F7C+JWkLomhhdM3TacNMjrzqLr3AkMlTQ4ez6M1D1RHHcHsD6wc0MFd3IyMysTVUeurQltA7xaYv84YKCkrgVxEyJiUYm4DqQmxJq4pRR1TWRxAFsXxFHi2sVxdXJyMjMrh7w34KbctIGk5wq2E9bwqr1J/UzFZmWPvXLG9S54nBO1+4NKxVHinMVxdfKACDOzMkg34eauFc34OP1bRZctddHiGcDWRhx1xObimpOZWblU59yazixK11Jqakyzc8bNKnjslY0CbCiOEufsXXS8Tk5OZmZloohcWxMax6r+n0JbA5MiYkFB3GBJxbNIbw0sY1Uf0zjSRATFU4HU9CG9VhBHiWsXx9XJycnMrBwa1+fUVB4CBkjatWaHpO7APtmxwrj2pFXBa+IqgUOBxyNiabb7UVKy+mbRdb5FmqN0QvZ8DGmCglJxs0grOdTLfU5mZmXR9HPrSToo+/Ez2eNXJU0HpkfE06SkMwa4U9KZrLoJV8BVK0sW8aKke4FrJbUn3Qd1Eunm3W8WxE3LZrw5V9J84AVSAtudgqHpEbFc0gWkm24/IN2EuztpNp0REbH6CpUlODmZmZVL0096cF/R81HZ49PAbhFRLWlv4OrsWCdSsvpiRLxf9NpjgB8Bl5GmL3oJ2DMiXiiKOw9YQJrBpmb6okMi4uHCoIi4UVKQpi86E5gEnBoRo8jBycnMrByi6Zdgj4gG192MiFmkGsuxDcQtBr6fbfXFVZES2GU5rn0TaQqjRnNyMjMrF08Xl5uTk5lZuTg35ebkZGZWJqpu4na9VszJycysHIKmvsG2VXNyMjMrA9HkN9i2ak5OZmbl4uSUm5OTmVm5ODnl5uRkZlYO7nNqFCcnM7My8Wi9/JyczMzKItys1whOTmZm5RA4OTWCk5OZWbm4VS83JyczszLxfU75OTmZmZWLk1NuTk5mZuUQAVVu18vLycnMrFxcc8rNycnMrFycnHJzcjIzK4cAqp2c8nJyMjMri4Bwn1NeTk5mZuUQeEBEIzg5mZmVi/uccnNyMjMrFyen3JyczMzKwhO/NoaTk5lZOQTgJTNyc3IyMysX15xyc3IyMysLT1/UGE5OZmblEBC+zyk3Jyczs3LxDBG5OTmZmZWL+5xyc3IyMyuHCI/WawQnJzOzcnHNKTcnJzOzsgiiqqq5C7HOcHIyMysHL5nRKE5OZmbl4qHkuTk5mZmVQQDhmlNuTk5mZuUQXmywMZyczMzKxAMi8lN4aOM6T9J04L3mLsdasgEwo7kLYbm15t/XphHRZ01fLOlR0ueTx4yI2HNNr9UaODlZiybpuYjYobnLYfn492VNpV1zF8DMzKyYk5OZmbU4Tk7W0t3c3AWwRvHvy5qE+5zMzKzFcc3JzMxaHCcnMzNrcZycDElHSwpJcyT1KjpWmR0b2QzlGibpWUkLszJsX+4ylNJcn0dTK/i9l9q+3ALK95Skp5q7HNY8PEOEFeoBnA2c09wFyfwaWAzsAywC3mze4rRaBwOTi/a91hwFMavh5GSFHgdGSLo2IqY2Z0EktQO2An4UEU80ENsxIpaWp2St0osR8XZDQf6crZzcrGeFLssez2soUNLnJI2WtCBrdvu7pM8VxdwmabKkT0v6p6RFkt6S9J0Gzn00UEX693lB1sw0seicwyQ9I2kxcFV27DBJT0ianpXrf5KOKjr3oOx8Rxft3y3bv1vBvgpJl0makpX9KUnbNPTZtAYFTX67SLpP0hzg2ezYZyXdn/0eFksaL+nHktYrOsdESbeVOHetZtHsd/eGpKWSxkk6YO29O1sXODlZoSnAdcAJkjatK0jStsDTQC/gaOBIoDvwtKTtisK7A3cBdwL7Af8FbpD0xXrK8Rdg5+znXwPDgMIvqx7APcDdwFez8wNsBtwPfBPYH3gYuKWhZFiPkcAPgd9l53sceGgNz9WSVWR9izVbRcGx3wETgINY1dw7EHgR+A6wJ/Bz4Fjg1jW5eNa/dRfwFvB14CfZObdak/NZ6+BmPSt2JXAicBHpC6eUC4GlwJciYg6ApL8BE7PXfb0gthtwckQ8mcX9A9gDOBx4stTJI2K6pNnZ08kRMbYopCvwrYj4U9Hrflzzc9Ys+BSwEXAScGNdb7iUbGDI6cDNEfGDbPfjkqqAKxpzrnXAG0XP/w3ckv18f0ScVXgwIh6o+VmSsvh5wO2STomImY28/sVZGfaLSGtKSHodGAuMb+S5rJVwzclWExGzgJ8CR0qq6y/XXYA/1ySm7HXzSLWKXYtiF9UkpixuKekv5IEfo5grgD8X75S0haS7JX0ALM+241izv8CHAl2A3xftv2cNztXSHQB8tmD7dsGxPxYHS+ou6UpJ75D+SFkO3AEI2KIxF85qaZ8lJcGVix1FxLOkP3asjXJyslKuAWYBl9RxvDepCbDYVFJTX6HZJeKWAp3WuHQwLSJWWxhHUlfgb8B2pOanL5C+9H4DdFyDa2yUPX5UtL/4eWvwakQ8V7AV1lZK/Z5vJTXp/QL4CulzPiU71tjf6wZAe0p/rq3xs7ac3KxntUTEAkmXk2pQPykRMgvYsMT+DbNja1upObeGAZsCX4iIf9XslFT8b3xJ9tihaP/6Rc9rvpT7AeMK9vdrXFHXeat91pI6kfoOR0bEzwv2Dy3x2iUUfc6SehfFzCDVvEp9rv1oveuUWQNcc7K6jAI+YNUIvkJPA3tJ6lazI/t5n+xYc+icPS6v2ZH1G+1XFPcRqeb2qaL9exU9fxlYCBxStP+wj1fMdV5HoIKCzzlzdInY96j9Oe9d+CSrAf8XOCjrJwRA0o7AoI9ZVluHueZkJUXEUkmXUHqW6UtJXzJ/l3Ql6a/rs0kJoq6mwLXtGVKn/PWSLiL1F51P+su8R01QRISke4FvS3qT1OG+F7Bb4ckiYo6ka4DzJM0njdQr7o9pcyJirqSxwBmSppA+32OBASXC7wF+k32OfyY1uR5dIu4i0uf7oKSbgD6kQRLNeq+dNS/XnKw+t5IGL6wmIl4mfZnPA35L6gxfAOwaES+Vs4AFZZpO6tivIA0nv5w04uzOEuGnAX8gDRW/l9RPMqJE3Ejgx8ARpMEee5Bqh23d4cDzwPXAbaQkclqJuN+yavTmw8BwVr8lAICIGE0a/r8V6fdyJvA9PFKvTfOSGWZm1uK45mRmZi2Ok5OZmbU4Tk5mZtbiODmZmVmL4+RkZmYtjpOTmZm1OE5O1mKUWDZ8vqSXJJ1aYhqipr52rXWesrWjJjbyPLtJGlk420ETlW+kpAbv+6hrDaW852+qz7ngdzmoKc5nbY+Tk7VEB5PmyjsQ+A/wS9IyHeV2KSVuGm3AbqQbT/1/y+xj8PRF1hIVLhv+uKQhpBkDSiYoSe2BFdHEd5RHxDtNeT4zy89/3dm64L9AN0l9C5rfTpZ0laQPSRO59gSQ9HVJY7Nl1edkS4yvtnaUpM6SRkmaqbSc+0PAxsUXLdWsJ6mLpCskvZMtKT5V0gOS+mVLj1+UhS6vaZ4suu6VkiZIWpY9nlfcBKhVy9ovkfSBpAtIayU1mqQ+km6S9Gb2mbwv6S5JpebCA/ikpCez2CmSLilRvg0k3ZCVbanS8uonrEn5zOrimpOtCwYDVaT5+2pmHz+PlLROIM2nt0RpOfYbSHMCXkJahXckafn4bSNifvbam4BDSZOL/pe0JlHNUu91ktSBtGbU9qS5+8aSJpUdTlrH6hZSkvs2aZn5qoLXVgKPAVuTmgtfAXYCLiCtj3VGFrcB8ARpvrqjSIn3TNZ8ccbepKUrzgWmA/2za/1b0iciYklR/IOkNbAuz97XBUA16XNEUnfSyrfrZfsmZHE3SOoYEb9cw3KarS4ivHlrERtpxuogTQBaSfrCP5H0Jf9gFjMoi3mBbG7IbH9XYC7wm6JzDgKWAd/Lnm+Vne+corgbsvMeXbDvNmBiwfNjs5h963kPI7OYyqL9R2T7dynaf15Wvr7Z8x9lzwcWxHQhzf4dOT7DicBt9RyvADbJynJAiXIXfy6/AuYDPbPnF5CS3RYl4mbUvO+C3+Wg5v535W3d3NysZy3RG6T1gmaR1pX6HSkxFHowIgr7mIYB3YHfSaqs2YDJ2fl2yeJ2JDVnr8ny63sAUyPioca8mcyepPWNnikq3+OklWB3KngfYyNiUs0LI2IhaVbvNSLppGzU4wLSEvc15y61fH2pz6Urq9Zl2hN4FphQ9D4eIy3YuPWaltOskJv1rCU6gJRU5gPvRe2mJ6i9fHjf7HF0HeesWS7+4yy/vj5pAcY10Ze0Um/xIn2F54ZUvldLHF+jJcsljSAtp/4zUvPgbFJyHkvpJdXr+lxq+qj6AkNo+H2YfSxOTtYSvRqrRuvVpXhk3szs8WhWX1a9Rk1/U+Hy6+8WHM+z/PoMaq/smtdMUv9M8cq6NSZmj1PqKMuaLg9/GPD3iDijZoekwfXE1/W51CTlmcA0Sq/fBF6DyZqIk5O1Fs+QEtCQiPhtPXHPkjr4DwGuKNifZ/n1x4HDJO0TEXU1sy3NHtdjVUIEeJR039aCiHijnmuMAc6UtElEvA9phCBrvshhZ9KikIWOqSe+1OeygFW1uUdJCzNOiohpa1gmswY5OVmrEBHzJJ1JWqa9D/AIaYDEAGBX4KmIuCsixku6C6gZIl0zWu9rOS5zJ3A8cLeky0mJrhtptNq1WdJ5LYs9Q9IjQFVEPEfqNzuGtLT9T4GXgA7A5sC+wP4RsQi4BjiZdH/XSFaN1lu8hh/No8DZkn5IuqF5d+CgeuKPL/hchgPHASMjYk52/BrSSMd/ZsuvjycN2PgE8IWI2G8Ny2m2GicnazUi4iZJ75O+zL9BGmjwAfAP4MWC0BNJtYEfkBLEE1n8vxo4/3JJe5DuZTohe5xJGlo9Kwv7M2kQx8mkm4ZFGlW4XNJw4JzstYOBhcA7wF9II/SIiBmSvgT8nLTM+UzgRtL/1TWZJeMS0j1gp5P6mJ4mJZ1364jfjzQjxwWk5H4Zaeh7zWcwV9Lns7KcTUr+c0hJ6oE1KJ9ZSV6m3czMWhwPJTczsxbHycnMzFocJyczM2txnJzMzKzFcXIyM7MWx8nJzMxaHCcnMzNrcZyczMysxfn/aiddLxTt1IUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(pipe_lr, X_valid_fold, y_valid_fold, display_labels=['Non fraud', 'Fraud']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Class weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Many sklearn classifiers have a parameter called `class_weight`.\n",
    "- This allows you to specify that one class is more important than another.\n",
    "- For example, maybe a false negative is 10x more problematic than a false positive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr_1_10 = Pipeline([\n",
    "    ('scale', StandardScaler()),\n",
    "    ('lr', LogisticRegression(max_iter=1000, class_weight={1:10}))\n",
    "]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Instead of decreasing the threshold, we boost up all the scores from `predict_proba` and keep the threshold at 0.5.\n",
    "- Either way we end up predicting more frauds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr_1_10.fit(X_train_fold, y_train_fold);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0sklEQVR4nO3dd5xdVbn/8c83nZBCOp0EwkVBildQuT+kWQLSRJBeAgKKgIh0IxKKUr2gV6pRULqABVSKoalIUESCRAgtFRJSJr1OZp7fH2tPcubkzMw+w+TMZOb7fr3268xZ+zl7r3NOMs+svdZeSxGBmZlZW9KptStgZmZWzMnJzMzaHCcnMzNrc5yczMyszXFyMjOzNqdLa1fAPryB/TvH0C26tnY1rAxvvtqztatgZVrEvDkRMai5rx+xz4Yxt6omV+w/X13xRETs19xztQdOTu3A0C268vcntmjtalgZRmy6S2tXwco0Nh6a8mFeP7eqhr8/sWWu2M6bvDXww5yrPXByMjOrgABqqW3taqw3nJzMzCogCKoj32U9c3IyM6sYt5zyc3IyM6uAIKjxdHG5OTmZmVVILU5OeTk5mZlVQAA1Tk65OTmZmVWIW075OTmZmVVAANXuc8rNycnMrAKC8GW9Mjg5mZlVQkCNc1NuTk5mZhWQZoiwvJyczMwqQtSg1q7EesPJycysAtKACCenvJyczMwqIN3n5OSUl5OTmVmF1LrllJuTk5lZBbjlVB4nJzOzCghEDZ1auxrrDScnM7MK8WW9/JyczMwqIBAro3NrV2O94eRkZlYB6SZcX9bLy8nJzKxCPCAiPycnM7MKiBA14ZZTXk5OZmYVUuuWU25OTmZmFZAGRPhXbl7+pMzMKsADIsrj5GRmViE1vs8pNycnM7MK8AwR5XFyMjOrkFqP1svNycnMrALSxK9OTnk5OZmZVUAgqj19UW5OTmZmFRCBb8Itgz8pM7OKELU5t2YdXXpcUki6sqi8n6QxkuZIWiJprKQdS7y+h6TrJM2QtEzSC5L2LBHXSdLFkiZLWi5pvKTDGqjTqZLekLRC0kRJX8/7fpyczMwqIEgtpzxbuSQdDexcolzAI8B+wFnAYUBX4BlJmxeF/ww4FfgecCAwA3hC0i5FcVcAo4GfAPsD44AHJX2x6NynArcBD2fnfxC4WdLped6TL+uZmVXIuhgQIWkj4AbgHODeot0HA3sA+0bEM1n8C8Ak4ALgm1nZzsAxwMkRcUdW9hwwAbg8Ow6SBgPnAVdHxPXZOZ6RNBy4GvhjFtcF+D5wV0SMKojbFLhC0piIqG7sfbnlZGZWAYGojXxbma4FJkTEfSX2HQy8X5eYACJiAfAocEhRXDXwQEHcKuB+YISk7lnxCKAbcHfRee4GdpQ0LHu+OzCoRNxdwABSwmyUk5OZWQUEUB1dcm15SdoDOAH4RgMhOwCvlSifAGwpqVdB3KSIWFoirhswvCBuBfB2iTiA7QviKHHu4rgG+bKemVlFqJz1nAZKeqng+e0RcXu9o0ldSX0610fExAaO0x+YXKK8KnvsByzO4uY1Ete/4HF+RESOOEocsziuQU5OZmYVEJQ1Q8SciNi1iZgLgQ1IfTsNUXbqUuXrOo4GYnNxcjIzq5CWWglX0pbAKOAUoHtBnxDZ842ARaSWSqlWSr/ssa5lUwVs2UhcVcFjP0kqaj2ViiM794yCuP5F+xvkPiczswqIELXRKdeWw9ZAD9KAg3kFG6TRdPOAHUl9PDuUeP32wNSIWJw9nwAMk9SzRNxK1vQxTQC6A9uUiAP4T0EcJc5dHNcgJyczswpIAyI659pyeAXYp8QGKWHtQ0oojwCbSdqr7oWS+gAHZfvqPEK6/+krBXFdgCOBJyNiRVb8OClZHVtUn+OA1yJiUvb8BWBOA3FVwPNNvUFf1jMzqwi12PRFETEfeHatM0gAUyLi2ez5I6REcbek80ktqotJfULXFhzvFUkPADdmAy0mAacDwyhIMBExS9INwMWSFgEvkxLYvhQMTY+IakmXkG66fQ8Ym8WcDJwVESubeo9OTmZmFZAGRFR2scGIqJV0IHA9cDPpUuALwD4RMa0o/CTS4IorgY2A8cB+EfFyUdwo0gi/s4GNgYnAERHxaNG5b5UUwLnA+cBU4MyIuDlP3Z2czMwqZF0vmRGxdvaLiCpSi+XkJl67DPh2tjUWV0NKYFc2FpfF3kYa7l42JyczswqomyHC8nFyMjOrkFqPQcvNycla1HeO2Zp/PtuHo8+eycgLZ64unzyxB7+8dmNef3lDlizsxJAtVjLiqCoOPWU2nbN/hR9M78rN392cdydswPy5XeixQS1DP7KcI874gN32XVTvPA/fNojxz/firVd7UjWrK8d9eybHnzeTUv54T38evnUwH0zrxpDNV3LoabM58IS56+wzaK/2OGA++3xpPtvuvJSNBqxi1vvdeP6Pfbn/x4NZtiSNMBu+41JOumgmQz+yjD79ali8sDNv/3sD7r1xCK//c8NWfgetKwKqa52c8nJyshbzzG82YtJ/NlirfO7MLlxw2HAGbFzN1y97j779V/Gvv/ZizBWbMn9OF075brpHb9mSzvTtv4oTL5zBwE2qWbqoE4/dO4DvHrcNl4yZxB5fXLD6mI/dM4CevWvYfb8F/OGXAxus0x/v6c+PL9iCI8/6gI9/ZjGv/LUXP7l4cyLgoBOdoMpx+NdnM/u9rtxx1SbMmdGV4R9bxnHnzmTn/1nMOQcPJ0L06lvD+5O78eSv+lH1QVc2GriKL582m+sefodzvzScia8U30bTcaTLek5OeVU0OUkaCdwBLACGRcS8gn1dSLPiXhYRoytcr92BG4GPAT2Bj0fEK5WsQynZSJeKfx7NsXhBZ24bvRlfG/0eV58xtN6+F8f2ZUFVF/73d2+x+Tbpdold9ljMjMndeeqh/quT09DtlvPt/60/gOhTn1vICZ/anifv718vOd3+7Bt06gQ1q2gwOdWsgjuv3oTPHj6Pky5Krapd/t9i5s7syi+v3YT9j5lLl64t9Qm0f5eeOIwFVWt+Zfx7XC8Wze/M+T+exk7/s5jxz/fmlb+mrdBLz/TmV69N4LOHV3Xo5AQtN0NER9BaabwvaV6otuJnpER9EGmq9zdbtzrrnzFXbsJW2y1nn0Pnr7WvemX6D9mzd0298g371lBb2/hxO3eBDfvU0KVr/Sm6OuX4l/uff27Igrld2ffL9WdK+ezh81g4rwuv/b1XA6+0UgoTU526ZDNw44aX5lm+tBPVK8Wq6o79i7luKPk6WDKjXWqt5PQkcJakjVvp/KtJ6gRsB/whIp6OiHElpo2vi+1eqryje+3FDRn7UH/O/MH0kvv3PGg+ffuv4qZRmzNzajeWLOrE84/15amH+nHY12avFV9bm1o9VbO6cM8NQ3jv3e4cNHJO2fWaMrEHAEM/srxe+dDt0vOpb/Yo+5hW3067p9lvpr5V/7OUgs5dgkGbreSM778HwOP3Dqh4/dqWFp2+qN1rrU+hbnz8qEajAEmfzNa8XyxpiaSnJH2yKOZOSdMlfVzSXyQtlfRWU+vVZ5cZa0ifwyWSQtLkomPuLulvkpaR3VEt6ShJT0uandXrX5JOLDr20Ox4I4vK987K9y4o6yzpSkkzsro/K6nUfFhtzqpq+NGFm3P412exxfAVJWP6DVrFDY++ydS3unPip7fny9vtxBWnDOWIM2ZxxBmz1oofc8WmfHHLXTh6l4/x4M2DufiWKXz8M4tLHLlxi+anTvrefeu32HpvtKrefmueARtXc8L5H/Dyn9PAlEKjbpvCH6e+yt3/eJ09DljAJccPWyuBdUS1KNdmrZecZpDWnz9N0lYNBUnaCXiONOPtSNKiWn2A57JlhQv1IS1RfDdpGo1/ALdI2oeG/YE1KzL+jHRJ79CC/X1JK0HeB+zPmiWQtwYeIk3r8SXSqpJjmkqGjRgNfAe4Jzvek9Sf96rN+tVNQ1i5vBNHf/ODBmPmz+3MFacMo0fPWr7700lc+9DbHH32B9z3oyE88JPBa8Ufeups/u+xiVz2i3fZdd+FXH3GVoz7U5+y67b6fkT/X29xPXrWMPqOSdSsgh+es8Va+8dcuQln7b8tl391Kya/0YPLfzGJbXcqeUGiw0ij9Trn2qx1R+tdA3wNuJSG71z+HmnVxc9mc0kh6U+kxbMuBb5cENsb+EbdcsSS/gx8ATgaeIYSImK2pLpBGdMjYlxRSC/guIj4XdHrflD3c3ZZ8FlgE9JcVLc29IZLkdQPOIe0mNh5WfGTkmqAqxt53WnAaQBbbtY6X+Os6V2578dDOOf6aVSv7ER1wWxZ1Ss6sXhBZzboVcODNw3hg2nd+OXf/0PvjVIrZuf/WUxtLfzyuo3Z7+i59B2wpnUzaNNqBm1aDSzj059fyPmHDeenl2/Kpz+/sKz6FbaQBgxZtbp80fwu2f6akq+zxnXtXstld05mky1Xct5h2zBnRre1YmZO7c7MqfDm+J68OLYPtz09kZEXzmTUsVu3Qo3bBt+EW55Wu7iZTanxQ+AESds1ELYn8Pu6xJS9biGpVbFXUezSusSUxa0A3qL0GiV5rQJ+X1woaVtJ92UTGlZn2ymkvqty7QhsCPyqqPz+xl4UEbdHxK4RseugAa3zl9aMqd1ZubwT15y5FYd9dMfVG8BDtw7msI/uyOTXN2DSGz3YdOiKtZLBdrssZVV1J96f3HhX3rY7LW0yppS6vqW6vqc6U7K+pi3/a/lar7HGde4SXPLTyWz38aV89/hhTH5j7VsHiq2q7sSk1zdg06GlL/t2JL6sl19r3+d0A3AWcDlrT60Oay9UVWcmaxa3qlNqieEVpIkOm2tWNo/UapJ6AX8ClgIXAe+QppA/nSbmrmrAJtlj8XWxhq+TtRHb7LCMax96e63yCw4fzmcPq2LE0VVsOmwF/Qev4vWXNmTR/M71EtQb/0o3ZQ5oZKRXbS1M+MeGbLpV+b/YPvqJJfTtv4qnf92P/95zTZ/V0w/3o3e/Veyw25Kyj9mRScFFP5nCx/dYzCUnDOONl/PdVNt9g1q23Xkp09/p2OOJWmPi1/VZqyaniFgs6SpSC+q6EiFVpFlvi21MjpUUW0CpJYZ3B7YCPhMRf60rzO7TKlT3Z3nxNY/iIUt1yXcIaxboqnvepvXqW8PO/1N6oMLgzVeu3nfA8XN4+tf9+M7R23D46bPo028Vr77Qi4dvHcT/238+gzdLyemu6zdm0fzObL/bEvoPrqZqVleeuK8/E//Vk4tumlLv+G+O34APpnWjtjb9Z5/yVnf+8vu+AOy270J69Ay6dIUTLpjBTy7enIGbVK++CfeJ+/vzjSvfo2u3Zq8g3SGd+YP32PPgBdx742CWL+3ER/57TXKfM6Mrc2Z045vXTGPR/C68NX4DFlR1YcjmKzn4pDn0H7yK6876MBcx2gePxMuvtVtOkKZx/zalZ7h9DjhAUu+IWAQgqTfpfqRnK1bD+uqGJa3+cz/rNzqkKO4DUsvtY0XlBxQ9fxVYAhwBPF1QftSHrmkb8dFPLOWHv3mLe27YmFu/txlLFqXpi4495wMO+9qa0XrDd1zKb8YM4tnfbcTSRZ3pN2gVW2+/jB/+5m12+GT9Vs4jdwziT79as/r0Xx7tx18eTY3pX7z4HzbumTrADjxhLlKa7uihWwYzaLNqzvj+dA4a6dkhyrXrvqnP75hvzeKYb9UfZXnXD4dw9w835o1/9WT/Y6r44rFz6dGzljkzuzLxXz3533O3yHUJsD2LEKucnHJr9eQUESskXQ7cXmL3FcCBwFOSriG1ZC4kJYjLK1fLev4GLARuknQpqb/ou6RVH/vWBUVEZIt3fVXSm6Q1Tw4A9i48WETMzxbvGpUt3vUksBvw1Qq8l3XiifdfWavso59YypV3v9vo63YfsZDdR+Qb9HDejVM578apuWIPOH4uBxzvZPRhnfip7ZuMefL+ATx5f0e/n6lhvqyXX1tJ43eQBi/UExGvkn6ZLwR+AdxFWuRqr4gYX8kKFtRpNmm4eWfScPKrgDGkIezFzgZ+TRoq/gCp/+usEnGjgR8Ax5MGe3yB1Do0s3bCM0SUp6Itp4i4E7izRPkq4L8aeM2LwOeaOO7IBsr3zlGnVZS4E6ahY2b7ngY+XmLX6KK4+aSEU0xFcTWk1td3G4szs/WbE09+rX5Zz8ysI/B9TuVxcjIzqxDfw5Sfk5OZWQVEwCovNpibk5OZWYX4sl5+Tk5mZhXgPqfyODmZmVVIODnl5uRkZlYhHhCRn5OTmVkFRLjPqRxOTmZmFSFqPFovNycnM7MKcZ9Tfg0mJ0m1lF4yopSICCc6M7MGeD2n8jSWUC4nf3IyM7PGROp3snwaTE4RMbqC9TAza/c8Wi+/si/FZcuUDwDej4iG19c2M7PVwgMiypL7k5J0oKSXgQXAu8COWfkYSceso/qZmbUbEfk2y5mcJH0J+B1ptdcLqb/O0CTgxBavmZlZOxOhXFtTJI2Q9LSkmZJWSJou6VeSti+K65c1IOZIWiJprKQdSxyvh6TrJM2QtEzSC5L2LBHXSdLFkiZLWi5pvKTDGqjjqZLeyOo3UdLXy/iocrecLgXuiIgvADcW7XsN+Fg5JzUz62hSq6hlkhPQH/gncCZp5eyLgR2AcZK2ApAk0sra+5FW4D4M6Ao8I2nzouP9DDgV+B5wIDADeELSLkVxV5AWVf0JsD8wDnhQ0hcLgySdCtwGPJyd/0HgZkmn53lzkL/P6aPABdnPxY3OeaQ+KDMza0RLDSWPiPuA+wrLJP0deAM4HPghcDCwB7BvRDyTxbxAutp1AfDNrGxn4Bjg5Ii4Iyt7DphAGrV9cFY2GDgPuDoirs9O+4yk4cDVwB+zuC7A94G7ImJUQdymwBWSxuQZr5C35bQQGNjAvqHA7JzHMTPrsNZxn9Pc7LHuF//BpIFrz6w5fywAHgUOKXjdwdlrHiiIWwXcD4yQ1D0rHgF0A+4uOu/dwI6ShmXPdwcGlYi7i9SQ2SPPm8mbnP4EXCxpo4KyyCp9JvBYzuOYmXVIgait7ZRry0tSZ0ndJG1Luow2k5RUIF3me63EyyYAW2Yjr+viJkXE0hJx3YDhBXErgLdLxAFsXxBHiXMXxzUq72W9UcDfgYmkplsAFwE7AX2BL+U8jplZh1VGo2igpJcKnt8eEbeXiHsR+ET289ukS3izsuf9gcklXlOVPfYDFmdx8xqJ61/wOD9irbZdqThKHLM4rlG5klNETJb038BlpKZdDbAn8DjwvYh4P89xzMw6rChrbr05EbFrjrjjgT7A1qT+oD9J2iMiJpNGVZfKh8WVWBdxNBCbW+6bcCNiOvDVD3MyM7MOrYXvYYqI17MfX5T0GKmldBHwdVJLpVQrpV/2WNeyqQK2bCSuquCxnyQVtZ5KxZGde0ZBXP+i/Y0q+3ZlSZtK2i0beWFmZjm14FDyEseO+aRLe3V9RBNY0/9TaHtgakQsLogbJqlnibiVrOljmgB0B7YpEQfwn4I4Spy7OK5R5cwQcYKkScA00tj2aZImSTou7zHMzDqqAGprlWtrDklDgI8A72RFjwCbSdqrIKYPcFC2j4K4rsBXCuK6AEcCT0bEiqz4cVKyOrbo1McBr0XEpOz5C6QJG0rFVQHP53k/uS7rSToT+DEwlnQT1gfAEOBo4BeS+kbETXmOZWbWIQXQQvc5SfoN8DLwKulWn/8CzgFWke5xgpR0XgDulnQ+6TLexaQ+oWtXVyviFUkPADdK6kq6D+p0YBgFCSYiZkm6gTRye1F2/iOBfSkYmh4R1ZIuId10+x4pb+wLnAycFREr87zHvH1O5wJ3RsTJReU/l3QnqSPOycnMrBEtOG/eOOAI0u/mbqQrWs8CV2WDIYiIWkkHAtcDNwM9SMlqn4iYVnS8k0g3zl4JbASMB/aLiJeL4kaRRvidDWxMGsF9REQ8WhgUEbdKiqx+5wNTgTMj4ua8bzBvctqYNWPni91L+pDMzKwxLZScIuIa4JoccVWkFktxw6I4bhnw7WxrLK6GlMCuzHHu20j3XjVL3uT0b9buBKuzLaVv9DIzs9WaP9ihI8qbnM4G7pc0B/h1RNRI6kyaSPB84Kh1VUEzs3bDy2Hk1mBykjSN+h9lX9KlvRpJ80hj2zuTrj8+AGy1DutpZrZ+C4hmjsTriBprOT2F87yZWQtycsqrweQUESMrWA8zs/bPf+7nlnv6IjMz+5CcnHIrKzlli1JtRxovX09E/LKlKmVm1u604E24HUHeGSI2Av4AfLquKHss/DvAycnMrBEteBNuu5d3br0fkFYw3JOUmA4lTUdxD/Au8Ml1Ujszs/akVvk2y52cRpAS1Ljs+fSIeDYiTiDNm3T2uqicmVl7osi3Wf7ktAnwbjZ1xXKgd8G+XwMHtHTFzMzalShjs9zJaSZpMkCAKcDuBfuGrxVtZmZFlAZE5Nks92i9v5IS0u+Bu4BLJQ0lTc9+IvXXBjEzs1LcKsotb3K6DKhb+fY60uCII4GepMR0VstXzcysnalt7QqsP3Ilp4h4h2x1xYioJq3Rce46rJeZWfvi+5zK4hkizMwqxCPx8mtsVvLvlXGciIgrWqA+Zmbtl5NTbo21nEaXcZwAnJzMzKxFNDYred5h5tbK3ny1JyM23aW1q2FmTfBlvfzc52RmVgmBpyYqg5OTmVmluOWUm5OTmVmF+LJefk5OZmaV4uSUm5OTmVmlODnl5uRkZlYBXg6jPOUu074TacHBAcBtETFT0nDgg4hYtC4qaGbWbni0Xm55l2nvDtwNfJm0Em4Aj5KW0rgWeBO4aB3V0cysXXDLKb+8N9p+H/gccDwwhJSg6jxGWinXzMwa48UGc8t7We9o4LsRca+kzkX7JgFDW7RWZmbtjfucypI3OQ0AXm9gXyege8tUx8ysHXNyyi3vZb1J1F+avdAngYktUx0zs/ZLtfk2y5+cfglcJOlYoFtWFpL2Ac4Bfr4uKmdmZh1T3uR0LfAH4C6gKiv7KzAWeDwi/m8d1M3MrH1poQERkg6X9LCkKZKWSZoo6SpJvYvi+kkaI2mOpCWSxkrascTxeki6TtKM7HgvSNqzRFwnSRdLmixpuaTxkg5roI6nSnpD0oqsfl9v+p2tkSs5RURNRBwF7AX8EBgD/BjYNyKOLeeEZmYdUqy5EbepLYfzgBrgO8B+wC3A6cCfJHUCkCTgkWz/WcBhQFfgGUmbFx3vZ8CpwPeAA4EZwBOSdimKu4K01t9PgP2BccCDkr5YGCTpVOA24OHs/A8CN0s6Pde7o8ybcCPiL8BfynmNmZllWm5AxEERMbvg+XOSqoBfAHsDTwMHA3uQGhHPAEh6gTSG4ALgm1nZzsAxwMkRcUdW9hwwAbg8Ow6SBpOS4tURcX123meyiRiuBv6YxXUh3X50V0SMKojbFLhC0piIqG7qDXpBQTOzSmmhy3pFianOP7LHzbLHg4H36xJT9roFpAkUDil43cFANfBAQdwq4H5gRDYJA6T7WbuRJmQodDewo6Rh2fPdgUEl4u4ijfzeo6n3BzmTk6RaSTWNbXmOY2bWUYl1Plpvr+yx7rafHYDXSsRNALaU1KsgblJELC0R1w0YXhC3Ani7RBzA9gVxlDh3cVyj8l7Wu5y18/kA4Auke5zuzHkcM7OOqbybcAdKeqng+e0RcXtDwZI2I/2eHhsRda/rD0wuEV43qK0fsDiLm9dIXP+Cx/kRUfwuSsVR4pjFcY3KlZwiYnSp8my2iEeBBXmOY2bWoeVPTnMiYtc8gVkL6HfAKuCkwl0NnLF49tl1EUcDsbl9qD6niKgBbga+9WGOY2bWIbTw3HqSepBG5G0NjIiI6QW7qyjdSumXPc7LGVdV8NgvGwXYVBwljtm/aH+jWmJARPcSlTAzsyItOJQcSV1JQ7U/CXwxIv5dFDKBNf0/hbYHpkbE4oK4YZJ6lohbyZo+pgmk3/fblIgD+E9BHCXOXRzXqLwDIrYssQ2X9CXSEMKXmjiEmZm13E24nYB7gM8Ch0TEuBJhjwCbSdqr4HV9gIOyfYVxXYGvFMR1AY4EnoyIFVnx46RkVXxv63HAaxExKXv+AjCngbgq4Pmm32H+ARGTafha4zvAGTmPY2bWMUWLzpt3EymZfB9YIunTBfumZ5f3HiElirslnU+6jHcx6ff2taurFfGKpAeAG7PW2CTSDb3DKEgwETFL0g3AxZIWAS+TEti+FAxNj4hqSZeQbrp9jzST0L7AycBZEbEyzxvMm5xOKlG2HJgC/CPrezIzs8a03E24+2ePo7Kt0GXA6IiolXQgcD1pbEAPUrLaJyKmFb3mJFKiuxLYCBgP7BcRLxfFjSKN8Dsb2Jg06fcREfFoYVBE3CopgHOB84GpwJkRcXPeN9hkcspG5L1Cupmr1I1fZmaWQ0ut5xQRQ3PGVZFaLCc3EbcM+Ha2NRZXQ0pgV+Y4922kKYyaJU+fU5D6lD7e3JOYmRleCbcMTbacsqbhNGDDCtTHzKx9cuIpS96h5LcB35LUrclIMzNbi2jZoeTtXd4BEb1JY9vflfQ4aTr1wo8wIuLSlq6cmVl74sSTX4PJSdK7wKERMZ60ZkidUh1rATg5mZk1xskpt8ZaTkNJdwMTEV5aw8zsw3Jyyq2sxQbNzKyZ3J9UlqaSkz9KM7OW4t+ouTWVnC6TNCfHcSIiTmyJCpmZtVctOH1Ru9dUctqFtPJhU/z3gJlZE3xZL7+mktOXIuLvFamJmVl75ptwy+IBEWZmleLklJuTk5lZBdTNEGH5ODmZmVWIap2d8mowOfnGWzOzFuQ+p7K45WRmViG+rJefk5OZWaU4OeXm5GRmViFuOeXn5GRmVilOTrk5OZmZVUJ4+qJyODmZmVWA73Mqj5OTtZqBm6zkiDNmse1Oy9h6h2X02CA44ZMf5YPp3VbHnHvDVL5w5LySr5/2dndO2fMjlaquZXbbdyFHnjmL4Tsuo7YW3nu3O2Ou3ITxz/f299WUcHbKq0MnJ0kjgTsa2P35iBhbweqsRdKzABGxd2vWY13ZdOhK9jxoAW+9ugGvvbghu+69eK2Ye28cwh/uGlCvbMgWK/nOLVN54ck+laqqZb543FzO+P50HrljIPfeOAQp2OZjy+mxQfql6++rcW455dehk1OBrwDTi8r+0xoV6Uj+PW5Djtp5BwD2O2ZuyeQ0Y0p3ZkzpXq/sv/dMcWN/1X/dV9JWG7L5Sr5+2XuMuWJTfjNm0Oryfz63Jun4+2qEb8Iti5NT8kpEvN1UkKTuEZFnCRHLIULNet3nDq/izfEbMOXNHi1cI2vMiKOriBC/L2oZNcXf1xoeEJGfpyhqgKSRkkLSnpIelDQfeDHbt5ukhyRNl7RM0kRJP5C0QdExJku6s8SxQ9LoorKjJL0haYWkCZIOXXfvbv21/W5L2GzrlfzpwX6tXZUOZ4fdljDt7e7sfch87vjb6/xx6njueP51DhrZ8Hqk/r7qU22+zdxyqtNZUuFnUdj4vge4DzicNZ/XlsArwJ3AImAH4HvA1sBR5Z5c0ueAe4E/AOcCg4AfAV2BieUerz373OFVVK8Uz/7Wv+wqbcDG1fQfUs0pl7zPnVdtwvtTurHngQs48wfv0blz8NufDVrrNf6+CgQeEFEGJ6fkjaLnzwNjsp8fiogLCndGxMN1P0tSFr8Q+KWkMyJibpnnvyyrwyERUZsd93VgHA0kJ0mnAacB9KBnmadbP3XtVsueBy3gxbF9WFjlf7qVJgUb9q7l8q9uxfOPbQTA+Od7M2SLlRx51ix++7OBpAHTib+vtXlARH6+rJccCuxWsH21YN9vioMl9ZF0jaR3SMvYVwN3kf5nblvOiSV1zs75UF1iAoiIF4HJDb0uIm6PiF0jYteudG8orF3ZfcRCem9Uw1hfImoVC+elBPPyn3vXK//nc73pP3gV/Yesqlfu76uEyLmZW06Z14oHREjaPftxRon4O4DPkS7lvQIsAT4J3ASU2+s7kHT57oMS+0qVdVif+0oV8+d25u9PeUhya5jyZg+233XpWuXKGktR1Ffi76s+34RbHrecmlbvn5OkHsAhwHUR8aOIeC4iXgKWlXjtcqBb0euLx9POIbW8hpR4famyDmmjgdV8Yq9FPPObftSsat4oP/tw/vZYXwA+sfeieuWf2Hshs9/vyrzZXVeX+fsqIQLV5tvMLafm6A50JiWUQiNLxE4BPlZUdmDhk4iokfQP4HBJowv6nD4FDM2O0W7tccB8ALbdMeX23fZdyPy5XVgwtwv/Htdrddy+X55Pl674ElEr+vtTvXnlr704+5rp9O2/ihlTu/OZA+az696Luf5bW9SL9ffVAOed3JycyhQRCySNA86VNIPU8jkZ2KxE+P3AzyXdAPwe2JnSSexS4Engt5JuI43WuwyY2fLvoG255Kf1c+9ZV78HwPi/bcgFhw9fXf75r1Qx6fUevP3vjjH4o20Sl508lJO+M4Pjz/uAXn1rmPZ2d64+Y0ue+U39JOTvq7SWvKwnaXPgQmBX0u+WDYBhETG5KK4fcB3wpSzmBeCciPh3UVwP4ArgOGAjUpfFhRHx56K4Ttl5vwZsTBq0dXnhQLGC2FNJI5CHkfrQb4iIW/O8Pyen5jkauIXUx7QM+BVwNikBFfoFsAVpgMXXgL+QBl/U69+KiLGSjgVGA7/O9n8rO2a7NmLTnXPFnf757dZxTSyPpYs7c9N3Nuem72zeaJy/rxICaNlLdsOBI4B/kn63fKE4IBtN/AgpOZwFzAMuBp6RtEtEFM6M8zPgAOB84F3gDOAJSbtHxCsFcVcA5wGjsnMfBTwo6cCI+GPBuU8FbgOuAsYCnwVulqSIuKWpN6fwuPv1Xh/1j0/ps61dDbN2bWw89M+I2LW5r+/dd/P47//3zVyxf37swibPJalTQTfAKcBPKWo5SToE+C2wb0Q8k5X1BSYBd0fEN7OynUktpZMj4o6srAswAZgYEQdnZYOBacDVEXFpwXmeAgZFxE4Fr30feCwiTiyI+zlwMLBJRBR3jdTjARFmZhWiyLflUXjrSSMOBt6vS0zZ6xYAj5IGdhXGVQMPFMStInVNjJBUd7/KCNIgr7uLznM3sKOkYdnz3UndE8VxdwEDgD2aqriTk5lZhbTCaL0dgNdKlE8AtpTUqyBuUkQU3yswgZSMhhfEraCoayKLA9i+II4S5y6Oa5CTk5lZJeS9ATflpoGSXirYTmvmWfuT+pmKVWWP/XLG9S94nB9r9weViqPEMYvjGuQBEWZmFZBuws3dKprzYfq3ik5b6qTFN5+tizgaiM3FLSczs0qpzbm1nCpKt1LqWkzzcsZVFTz2y0YBNhVHiWP2L9rfICcnM7MKUUSurQVNYE3/T6HtgakRsbggbpik4hvTtgdWsqaPaQJpIoJtSsTBmkVa6/qWis9dHNcgJyczs0oor8+ppTwCbCZpr7oCSX2Ag7J9hXFdSauC18V1AY4EnixYZPVxUrI6tug8x5HmKJ2UPX+BNEFBqbgq0koOjXKfk5lZRbT8vHmSDs9+/ET2uL+k2cDsiHiOlHReAO6WdD5rbsIVcO3qmkW8IukB4EZJXUn3QZ1Ounn32IK4WdmMNxdLWgS8TEpg+1IwND0iqiVdQrrp9j3STbj7kmbTOSsiVjb13pyczMwqpeUnPXiw6PnN2eNzwN4RUSvpQOD6bF8PUrLaJyKmFb32JOD7wJWk6YvGA/tFxMtFcaOAxaQZbOqmLzoiIh4tDIqIWyUFafqi84GpwJkRcTM5ODmZmVVCtPwS7BHR5JTvEVFFarGc3ETcMuDb2dZYXA0pgV2Z49y3kaYwKpuTk5lZpXi6uNycnMzMKsW5KTcnJzOzClFtC1/Xa8ecnMzMKiFo6Rts2zUnJzOzChAtfoNtu+bkZGZWKU5OuTk5mZlVipNTbk5OZmaV4D6nsjg5mZlViEfr5efkZGZWEeHLemVwcjIzq4TAyakMTk5mZpXiq3q5OTmZmVWI73PKz8nJzKxSnJxyc3IyM6uECKjxdb28nJzMzCrFLafcnJzMzCrFySk3Jyczs0oIoNbJKS8nJzOziggI9znl5eRkZlYJgQdElMHJycysUtznlJuTk5lZpTg55ebkZGZWEZ74tRxOTmZmlRCAl8zIzcnJzKxS3HLKzcnJzKwiPH1ROZyczMwqISB8n1NuTk5mZpXiGSJyc3IyM6sU9znl5uRkZlYJER6tVwYnJzOzSnHLKTcnJzOzigiipqa1K7HecHIyM6sEL5lRFicnM7NK8VDy3JyczMwqIIBwyyk3Jyczs0oILzZYDicnM7MK8YCI/BQe2rjekzQbmNLa9VhHBgJzWrsSllt7/r62iohBzX2xpMdJn08ecyJiv+aeqz1wcrI2TdJLEbFra9fD8vH3ZS2lU2tXwMzMrJiTk5mZtTlOTtbW3d7aFbCy+PuyFuE+JzMza3PccjIzszbHycnMzNocJydD0khJIWm+pH5F+7pk+0a3Qr12l/SipCVZHXapdB1Kaa3Po6UVfO+lts+1gfo9K+nZ1q6HtQ7PEGGF+gIXAhe1dkUyPwOWAQcBS4E3W7c67dZXgOlFZf9pjYqY1XFyskJPAmdJujEiZrZmRSR1ArYDvh8RTzcR2z0iVlSmZu3SKxHxdlNB/pytknxZzwpdmT2OaipQ0icljZW0OLvs9pSkTxbF3ClpuqSPS/qLpKWS3pL09SaOPRKoIf37vCS7zDS56Ji7S/qbpGXAtdm+oyQ9LWl2Vq9/STqx6NhDs+ONLCrfOyvfu6Css6QrJc3I6v6spB2a+mzag4JLfntKelDSfODFbN9ukh7KvodlkiZK+oGkDYqOMVnSnSWOvdZl0ey7e0PSCkkTJB267t6drQ+cnKzQDOAnwGmStmooSNJOwHNAP2AkcALQB3hO0s5F4X2Ae4G7gUOAfwC3SNqnkXr8Adgj+/lnwO5A4S+rvsD9wH3A/tnxAbYGHgKOBb4EPAqMaSoZNmI08B3gnux4TwKPNPNYbVnnrG+xbutcsO8eYBJwOGsu924JvAJ8HdgP+BFwMnBHc06e9W/dC7wFfBm4Ljvmds05nrUPvqxnxa4BvgZcSvqFU8r3gBXAZyNiPoCkPwGTs9d9uSC2N/CNiHgmi/sz8AXgaOCZUgePiNmS5mVPp0fEuKKQXsBxEfG7otf9oO7n7LLgs8AmwOnArQ294VKygSHnALdHxHlZ8ZOSaoCryznWeuCNoufPA2Oynx+KiAsKd0bEw3U/S1IWvxD4paQzImJumee/LKvDIRFpTQlJrwPjgIllHsvaCbecrJ6IqAJ+CJwgqaG/XPcEfl+XmLLXLSS1KvYqil1al5iyuBWkv5C3/BDVXAX8vrhQ0raS7pP0HlCdbafQvL/AdwQ2BH5VVH5/M47V1h0K7FawfbVg32+KgyX1kXSNpHdIf6RUA3cBArYt58RZK203UhJcvdhRRLxI+mPHOignJyvlBqAKuLyB/f1JlwCLzSRd6is0r0TcCqBHs2sHsyKi3sI4knoBfwJ2Jl1++gzpl97Pge7NOMcm2eMHReXFz9uD1yLipYKtsLVS6nu+g3RJ78fA50mf8xnZvnK/14FAV0p/ru3xs7acfFnP1hIRiyVdRWpBXVcipArYuET5xtm+da3UnFu7A1sBn4mIv9YVSir+N748e+xWVD6g6HndL+UhwISC8iHlVXW9V++zltSD1Hc4OiJ+VFC+Y4nXLqfoc5bUvyhmDqnlVepzHUL7XafMmuCWkzXkZuA91ozgK/QccICk3nUF2c8HZftaQ8/ssbquIOs3OqQo7gNSy+1jReUHFD1/FVgCHFFUftSHq+Z6rzvQmYLPOTOyROwU1v6cDyx8krWA/wEcnvUTAiDpU8DQD1lXW4+55WQlRcQKSZdTepbpK0i/ZJ6SdA3pr+sLSQmioUuB69rfSJ3yN0m6lNRf9F3SX+Z964IiIiQ9AHxV0pukDvcDgL0LDxYR8yXdAIyStIg0Uq+4P6bDiYgFksYB50qaQfp8TwY2KxF+P/Dz7HP8PemS68gScZeSPt/fSroNGEQaJNGq99pZ63LLyRpzB2nwQj0R8Srpl/lC4BekzvDFwF4RMb6SFSyo02xSx35n0nDyq0gjzu4uEX428GvSUPEHSP0kZ5WIGw38ADieNNjjC6TWYUd3NPBP4CbgTlISObtE3C9YM3rzUWAE9W8JACAixpKG/29H+l7OB76FR+p1aF4yw8zM2hy3nMzMrM1xcjIzszbHycnMzNocJyczM2tznJzMzKzNcXIyM7M2x8nJ2owSy4YvkjRe0pklpiFq6XOvtc5TtnbU5DKPs7ek0YWzHbRQ/UZLavK+j4bWUMp7/Jb6nAu+y6EtcTzreJycrC36CmmuvMOAvwP/R1qmo9KuoMRNo03Ym3Tjqf9vmX0Inr7I2qLCZcOflDScNGNAyQQlqSuwKlr4jvKIeKclj2dm+fmvO1sf/APoLWlwweW3b0i6VtL7pIlcNwKQ9GVJ47Jl1ednS4zXWztKUk9JN0uaq7Sc+yPA5sUnLXVZT9KGkq6W9E62pPhMSQ9LGpItPX5pFlpdd3my6LzXSJokaWX2OKr4EqDWLGu/XNJ7ki4hrZVUNkmDJN0m6c3sM5km6V5JpebCA/iopGey2BmSLi9Rv4GSbsnqtkJpefXTmlM/s4a45WTrg2FADWn+vrrZx0eRktZppPn0listx34LaU7Ay0mr8I4mLR+/U0Qsyl57G3AkaXLRf5DWJKpb6r1BkrqR1ozahTR33zjSpLIjSOtYjSElua+SlpmvKXhtF+AJYHvS5cJ/A58GLiGtj3VuFjcQeJo0X92JpMR7Ps1fnLE/aemKi4HZwKbZuZ6X9JGIWF4U/1vSGlhXZe/rEqCW9DkiqQ9p5dsNsrJJWdwtkrpHxP81s55m9UWEN29tYiPNWB2kCUC7kH7hf430S/63WczQLOZlsrkhs/JewALg50XHHAqsBL6VPd8uO95FRXG3ZMcdWVB2JzC54PnJWczBjbyH0VlMl6Ly47PyPYvKR2X1G5w9/372fMuCmA1Js39Hjs9wMnBnI/s7A1tkdTm0RL2LP5efAouAjbLnl5CS3bYl4ubUve+C73Joa/+78rZ+br6sZ23RG6T1gqpI60rdQ0oMhX4bEYV9TLsDfYB7JHWp24Dp2fH2zOI+Rbqc3Zzl178AzIyIR8p5M5n9SOsb/a2ofk+SVoL9dMH7GBcRU+teGBFLSLN6N4uk07NRj4tJS9zXHbvU8vWlPpderFmXaT/gRWBS0ft4grRg4/bNradZIV/Ws7boUFJSWQRMibUvPcHay4cPzh7HNnDMuuXiP8zy6wNICzA2x2DSSr3Fi/QVHhtS/V4rsb9ZS5ZLOou0nPr/ki4PziMl53GUXlK9oc+lro9qMDCcpt+H2Yfi5GRt0WuxZrReQ4pH5s3NHkdSf1n1OnX9TYXLr79bsD/P8utzWHtl17zmkvpnilfWrTM5e5zRQF2auzz8UcBTEXFuXYGkYY3EN/S51CXlucAsSq/fBF6DyVqIk5O1F38jJaDhEfGLRuJeJHXwHwFcXVCeZ/n1J4GjJB0UEQ1dZluRPW7AmoQI8Djpvq3FEfFGI+d4AThf0hYRMQ3SCEGav8hhT9KikIVOaiS+1OeymDWtucdJCzNOjYhZzayTWZOcnKxdiIiFks4nLdM+CHiMNEBiM2Av4NmIuDciJkq6F6gbIl03Wu+LOU5zN3AqcJ+kq0iJrjdptNqNWdL5TxZ7rqTHgJqIeInUb3YSaWn7HwLjgW7ANsDBwJciYilwA/AN0v1do1kzWm9ZMz+ax4ELJX2HdEPzvsDhjcSfWvC5jABOAUZHxPxs/w2kkY5/yZZfn0gasPER4DMRcUgz62lWj5OTtRsRcZukaaRf5seQBhq8B/wZeKUg9Guk1sB5pATxdBb/1yaOXy3pC6R7mU7LHueShlZXZWG/Jw3i+AbppmGRRhVWSxoBXJS9dhiwBHgH+ANphB4RMUfSZ4EfkZY5nwvcSvq/2pxZMi4n3QN2DqmP6TlS0nm3gfhDSDNyXEJK7leShr7XfQYLJP1PVpcLScl/PilJPdyM+pmV5GXazcyszfFQcjMza3OcnMzMrM1xcjIzszbHycnMzNocJyczM2tznJzMzKzNcXIyM7M2x8nJzMzanP8PjLL7Uc6O1jwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(pipe_lr_1_10, X_valid_fold, y_valid_fold, display_labels=['Non fraud', 'Fraud']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Notice we've predicted more Fraud this time (see the right column).\n",
    "- This was like to saying \"repeat every positive example 10x in the training set\".\n",
    "  - But repeating data would slow down the code, whereas this doesn't.\n",
    "  - There are some fancier methods like SMOTE: see [paper](https://arxiv.org/pdf/1106.1813.pdf) and [code](https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html)\n",
    "- A useful setting is `class_weight=\"balanced\"`.\n",
    "  - This sets the weights so that the classes are \"equal\".\n",
    "  - You could also achieve this by setting the class weight manually by looking at the imbalance ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr_balanced = Pipeline([\n",
    "    ('scale', StandardScaler()),\n",
    "    ('lr', LogisticRegression(max_iter=1000, class_weight='balanced'))\n",
    "]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr_balanced.fit(X_train_fold, y_train_fold);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEQCAYAAAAUILtVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA20ElEQVR4nO3dd5wV1fnH8c9Dl97BRolYYjexhPyMYokl9thjQ43GGmMvxIi9JmgKisGoEVvUaNRERRTUKFgDKipYQIr0pS1lWXaf3x9nFmZnZ3fnLsvdZff7fr3mdfeeee7MuXdhn3vOnDnH3B0REZH6pEldV0BERCRJyUlEROodJScREal3lJxERKTeUXISEZF6p1ldV0DWXdfOTb3P5s3ruhqSg8mftKnrKkiOlnrBfHfvVtPXH7hPG19QUJIp9sOPi15x94Nqeq6GQMmpAeizeXPee2Xzuq6G5OCgvnvUdRUkR6+ufPTbdXn9goIS3nulV6bYpht/2XVdztUQKDmJiOSBA6WU1nU1NhhKTiIieeA4xZ6tW0+UnERE8kYtp+yUnERE8sBxSjRdXGZKTiIieVKKklNWSk4iInngQImSU2ZKTiIieaKWU3ZKTiIieeBAsa45ZabkJCKSB46rWy8HSk4iIvngUKLclJmSk4hIHoQZIiQrJScRkbwwSrC6rsQGQ8lJRCQPwoAIJaeslJxERPIg3Oek5JSVkpOISJ6UquWUmZKTiEgeqOWUGyUnEZE8cIwSmtR1NTYYSk4iInmibr3slJxERPLAMVZ507quxgZDyUlEJA/CTbjq1stKyUlEJE80ICI7JScRkTxwN0pcLaeslJxERPKkVC2nzJScRETyIAyI0J/crPRJiYjkgQZE5EbJSUQkT0p0n1NmSk4iInmgGSJyo09KRCRPSr1Jpq0mzOxlM3MzuylR3snMhpvZfDNbZmajzGyHlNe3MrM7zWyWma0ws7FmtldKXBMzu9rMpprZSjObYGZHV1Kns8zsCzMrMrNJZnZO1vej5CQikgdh4tcmmbZcmdmJwE4p5QY8DxwEXAgcDTQHRpvZZonwB4CzgN8BhwKzgFfMbOdE3I3AYODPwMHAOOApM/tZ4txnAcOAZ6LzPwUMNbNzs7wndeuJiOSBYxSvh+mLzKwjMAS4GHgssftwYE9gX3cfHcWPBaYAVwC/jsp2An4BnOHuD0ZlbwATgRui42Bm3YHLgNvc/a7oHKPNrB9wG/CfKK4ZcDPwiLsPisVtAtxoZsPdvbiq96WWk4hIHrhDiTfJtOXoDmCiuz+esu9w4LuyxBTq4YuBF4AjEnHFwJOxuNXAE8CBZtYyKj4QaAGMSJxnBLCDmfWNnvcHuqXEPQJ0ISTMKik5iYjkhVGacct8RLM9gVOB8yoJ2Q74NKV8ItDLzNrG4qa4+/KUuBZAv1hcEfBVShzAtrE4Us6djKuUuvVERPLAIZdWUVcz+yD2/H53vz8eYGbNCdd07nL3SZUcpzMwNaW8IHrsBBRGcQuriOsce1zk7p4hjpRjJuMqpeQkIpInOQx2mO/uu1YTcyWwEeHaTmWMkBfTytd3HJXEZqLkJCKSB47V2mKDZtYLGAT8EmgZuyZE9LwjsJTQUklrpXSKHstaNgVAryriCmKPnczMEq2ntDiic8+KxXVO7K+UrjmJiOSBA8XeLNOWwfeAVoQBBwtjG4TRdAuBHQjXeLZLef22wDR3L4yeTwT6mlnrlLhVrL3GNBFoCWyREgfwWSyOlHMn4yql5CQikhdGScYtg/HAPikbhIS1DyGhPA9samZ7r6mFWXvgsGhfmecJ9z8dG4trBhwPjHT3oqj4ZUKyOilRn5OBT919SvR8LDC/krgC4O3q3qC69URE8sChxrM/VDiW+yJgTLI83HPLt+4+Jnr+PCFRjDCzywktqqsJ14TuiB1vvJk9CdwdDbSYApwL9CWWYNx9rpkNAa42s6XAR4QEti+xoenuXmxm1xJuup0JjIpizgAudPdV1b1HJScRkTzJ90q47l5qZocCdwFDCV2BY4F93H16Ivx0wuCKm4COwATgIHf/KBE3iDDC7yKgJzAJOM7dX0ic+z4zc+BS4HJgGnCBuw/NUnclJxGRPHC3Wms5VX6OiiMu3L2A0GI5o5rXrgAuibaq4koICeymquKi2GGE4e45U3ISEcmDMCCi9qcvaqiUnERE8sJqMjVRo6XkJCKSB2FAhBYbzErJSUQkT7TYYHZKTiIieVCbM0Q0BkpOIiJ5UqqWU2ZKTlKrrvnF9/hwTHtOvGg2A6+cDcBdv+nFq/9In4R4sy1W8sBbXwAwecJG/GdEFz4Z15Z5M1vQvvNqtt+jkIFXzqZnr/L37C0paMqjQ3oy7tX2FMxtTqduxey+3xJOvnQ2HbuUrIl79R+dGDuyA19OaM3cmS346XEFXHb3tPX07jdcXXuu4thzvmOrHZbR9/sraLVRKaftuRNzZrYsF9e2/Wp+ec10+v90IS1blfL5R20ZdlMvpk4qP+vNy1PeSz3PeT/bjm8+bwPApn1XcNgpc9mp/xJ6bl7EimVNmfxxGx7+w2ZM+Tw5i86Gzx2KS5WcslJykloz+tmOTPlsowrlv/jNbA45ZX65sjnTW3DreX3of8DiNWVjnuvEt5NaceSZ8+i99Urmz2rOY3f35IKDt2LoyEl03zQsnOkO1w3sy8xvWnHK5bPotWUR0ya35OE7N+arT1oz5Pkvsaj35LVnOrN4QTN+sNdS3nyx43p77xu6TXqvZK9DCvjykzZMfL8tP9xrSUqUM3j4ZHpsVsS9g3uzdHFTjj9vFrc/9gXnH7I982e3KBc98qmu/Oex7uXKZk5ptebnH/xkCTv1X8Krz3Tlq0/b0Lb9ao791SzueXYilxyzLV992mZ9vNU6E7r1lJyyymtyMrOBwIPAYqCvuy+M7WtGWInxencfnOd69QfuBrYHWgO7uPv4fNYhTXR3dd4/j5ooXNyUYYM35VeDZ3Lb+X3K7dukzyo26VO+5fPRm+0A2P+4tcu9HHfBnHKtHoDtdl/GaXtsy0uPduG0K0JLbOY3Lfnsg7ZcdMd0fnbyAgB2+nEh1gT+dNXmzPi6JZv3C1OB3fL41zSJ/h58MKZdrb3fhuaT99px4m4/AOCg4+emJqcf/XQR2+9WyBUnbsPH49oD8Pn/2vLwmxM49lezuPf63uXiF8xpwRfj21Y4Tpk3XujMC3/vTny1hfFj2/PwWxM48vTZ3HVpcm7RDV++Z4jYkNVVGu9AWIukvniAkKgPIywvPLluq7PhGX7TxvTeeiX7HLUoU/yopzuz5Y7L6bP1yjVlycQE0GOzYjp0Wc2C2c3XlK0uDv/BW7ctH9+2fXgen8i/ib6oZpIysUAF/fdbyPzZzdckJoDlS5sx7rWO/OinaevUVW3JwuYklwFavrQZM6e0okuP4pyPV9+VDSXPskndJaeRwIVm1rOOzr+GmTUBtgb+7e6vu/u4lKWKy2JbppU3dp++24ZRT3fmgltmZIqf+F4bvpvSkp8eW+2SLkz7siWL5jdn8y3XJrHeW69khx8V8ujdPZg8YSNWLGvCF/9rzaNDerLbvkvotWVRFUeUmuq11Qq+nVyx2/bbL1vTY9NVtGpd/svCISfN4fkv3ue5zz7gtkc/Z7vdllZ7jrYdVtNnqxVM/7pVtbEbntCtl2WTuktOZXMyDaou0Mx2N7NRZlZoZsvM7DUz2z0R85CZzTCzXczsLTNbbmZfmtk51Rx7IFBC+ByuNTM3s6mJY/Y3s3fMbAXRLL5mdoKZvW5m86J6/c/MTkscu090vIGJ8gFR+YBYWVMzu8nMZkV1H2NmaWuw1Duri+GeKzfjmHPmrulKq86opzvRrHkpA46s+tt2yWr445Wb06FLMQeduDaRmcFNI75h8y2KuPDgrTlyyx256JCt6Nm7iGv/OqWKI8q6aNdxNUsXV7wSULgoTMnTtsPqNWWvPduFP1/bh6tP2YY/XtOH9p1Wc/ujX7DjHmnXstY6b/C3YPDs3+r8e+t6UYpl2qTuktMs4M/A2WbWu7IgM9sReIOwyuJA4FSgPfCGme2UCG8PPEZYy+QI4H3gXjPbh8r9G9gz+vkBQpfeUbH9HYAngMeBg6PjQ1jo62nCVPJHAi8Aw6tLhlUYDFwDPBodbyTl11qpt/7xlx6sWtmEE389J1P8qiLjzRc6ssf+S+iQ0o0X9+dBm/HZB2248k/TaNexfOyQyzbn84/a8Ovbp3PXP7/k17dP58sJrbnp7D6Ultb47UgVDLIt0A3ceckWvPnvLkx8vx2vP9eVS4/dlgVzm3PapZW3ro8/9zv2PXIBQ6/rzaxvG17LKYzWa5ppk7odrXc78CvgOiqfLfd3QBGwX7R+CWb2KjA1et3PY7HtgPPcfXQU9yZwAHAiMDrt4O4+z8zKvr7PcPdxiZC2wMnu/q/E624p+znqFhwDbExY/+S+yt5wGjPrBFwM3O/ul0XFI82sBLititedDZwN0GvTuvk1zp3RnMf/2IOL75pO8aomFMfGPBQXNaFwcVM2altC09j/tbGvdKBwcTP2r6ZL72+3bMxLI7pw2T3T+OGA8t1B745qz5jnOnHbk1+xy0/CQp47/GgZPXsVcc2J/Rg3sj0/Pqjqb+iSu6WLm9Gu4+oK5W07hC8OhSmtqjIrljXlvdc7cuBx81L3/+wXczn9ihk8dNdmjHyqW+1UuJ7RTbi5qbPOzWga998Dp5rZ1pWE7QW8WJaYotctIbQq9k7ELi9LTFFcEfAl0GsdqrkaeDFZaGZbmtnj0SJaxdH2S8K1q1ztALQB/pEof6KqF7n7/e6+q7vv2q1L3XzTmjWtJatWNuH2C3pz9Pd3WLMBPH1fd47+/g5M/bz8NYpRT3WiQ+fV7L5f5cnjsXt68OSfe3DuDTPZ/5iKXX9TPw/fqrfaufylwW12Cc+nf9nwvnXXB99O3oheW66oUN673wrmzGzByuVV/zu0Sv4u73fUfC64cSpP/7UnT/xlk9qoar2lbr3s6vo+pyHAhcANVFzOF6AzoQswaTahqy8u7QJGEWFxrZqaG61dsoaZtQVeBZYDVwFfE5YtPpdq1kupxMbRY7JfLFs/WR3aYrsV3PH0VxXKrzimH/sdXcCBJxawSd+116EWzmvGh2+057DT5tOseYWXAfDc8K48fPvGDLzqO444c35qTKfu4dv7pP+15gd7Fa4p/+KjcF9Ml40b3kiv+mDcqE4ceNx8dthjCZ+8G0bstW5bwh77LWT0812qfG3rtiXsvs+iCkPLf3xAAZfc8Q0vP9mN4besy/fI+k8Tv+amTpOTuxea2a2EFtSdKSEFhJUWk3pG+9a3tB72/kBv4Cfu/t+ywug+rbiy4WUtEuXJ/8VlybcHMDFW3iO3quZf2w4l7PTjwtR93TdbVWHf6//sRMlqq7RLb8xzHbnvuk3ZdZ8l7Px/hXz+4dpZAlq3K6H3ViHR/d/PFvHQ7T2586Le/OI3s9m8XxHTv2rJo3/oSbdNVvF/B6+9sffbyS2ZNjl8Pyla2YQ5M5rz1osdANihf2Hq8PXGas+Dw++l3/ahBbrrgEUsLmjO4oJmfPJue8aN6shnH7blij98w/BbN6dwSVOOP3cWZvDUsI3XHOfos2ax2fdWMmFsOwrmtKD7pkUcfdZsOnUr5vaL1967tP3uS7jqj18z5YvWvPp0V7bZee2/l+JVxtefNaybcKH2lmlvDOq65QRh6eBLSF9V8Q3gEDNr5+5LAcysHeF+pDF5q2F5ZX8x13w9j64bHZGIm0NouW2fKD8k8fxjYBlwHPB6rPyEda5pPfPqU53ps80KttyxYtcQwAdj2uNufDC6PR+Mbl9u3479C7nzmdBKa9OulLtf+JIRv+/JU0O7UzC3OZ27F7PHT5dwyqWz2ajN2hERbz7fiRF/WPv95uN32vHxO+Fm3Due/oqOlSTXxui3Q8u3gi+86VsAPh7XjitODL+b3525FWddM40LbpxK85bO5x+15YpffJ/5s9beZTHjm1b8+ICF/PiAhbRpV8LywiZ89mE7hlzVl8kT1racdu6/hBYtnX7bL2fIM5+XO/ecGS047Sc7r783WwfcjdVKTpnVeXJy9yIzuwG4P2X3jcChwGtmdjuhJXMlIUHckL9alvMOsAT4i5ldR7he9FtgPmF0HwDu7mb2JHCmmU0GJhES04D4wdx9kZkNAQaZ2VLCSL3dgDPz8F7Wi1e+G59aft+oSVW+7rK7p2We9677psVc8ofp1cadctlsTrlsdqZjNnYH9d292pjCxc0YcuX3GFLFLfTvvtaJd19L9rpXNOKezRhxz2a5VHGDp2697OpLGn+QMHihHHf/mPDHfAnwMPAIUAjs7e4T8lnBWJ3mEYabNyUMJ78VGE4Ywp50EfBPwlDxJwnXvy5MiRsM3AKcQhjscQChdSgiDYRmiMhNXltO7v4Q8FBK+Wpgq0pe8y6wfzXHHVhJ+YAMdVpNyp0alR0z2vc6sEvKrsGJuEWEhJNkibgSQuvrt1XFiciGTYknuzrv1hMRaQx0n1NulJxERPJE9zBlp+QkIpIH7rBaiw1mpuQkIpIn6tbLTslJRCQPdM0pN0pOIiJ5kmVRRwmUnERE8kQDIrJTchIRyQN3XXPKhZKTiEheGCUarZeZkpOISJ7omlN2lSYnMyslfcmINO7uSnQiIpXQek65qSqh3ED25CQiIlXxcN1Jsqk0Obn74DzWQ0Skwaut0XpmdiBh+aBtCauCzyMs5zPY3T+LxXUiLOR6JLARMBa42N0/SRyvFWGJopOBjsB44Ep3fzMR1yQ6768Ii75OAm5w92dS6ngWcCnQF5gKDHH3+7K+x5yvzplZWzPrbWaVLLQtIiJJHg2IyLJl0Bn4ELiAsMTO1cB2wDgz6w1gZkZYgucgwlI9RwPNgdFmllxI6wHgLOB3hDX0ZgGvmNnOibgbCasv/Bk4GBgHPGVmP4sHRYlpGPBMdP6ngKFmdm6WNwc5DIgws0MJXX07RUW7AR+Z2XDgdXd/LOuxREQao9rq1nP3x4HH42Vm9h7wBXAM8HvgcGBPYF93Hx3FjAWmAFcAv47KdgJ+AZzh7g9GZW8AEwl/8w+PyroDlwG3uftd0WlHm1k/4DbgP1FcM+Bm4BF3HxSL2wS40cyGu/ualcQrkylFm9mRwL8Iq71eSfl1hqYAp2U5johIY+ZumbYaWhA9lv3hPxz4riwxhfP7YuAF4IjY6w6PXvNkLG418ARwoJm1jIoPBFpQcWHVEcAOZtY3et4f6JYS9wjQhZAwq5W1W+864EF3PwC4O7HvU2D7jMcREWmU3Gs/OZlZUzNrYWZbErrRZhOSCoRuvk9TXjYR6GVmbWNxU9x9eUpcC6BfLK4I+ColDsL1r7I4Us6djKtS1uT0fdZm1WTDdCEhG4qISBVyWKa9q5l9ENvOruSQ7xISxmRgR0IX3txoX2fC3+ekguixU8a4zrHHRe4VOifT4kg5ZjKuSlmvOS0Bulayrw9hpIiIiFQhh2tO89191wxxpwDtge8Rrge9amZ7uvtUwuWXtDMmm2brI45KYjPL2nJ6FbjazDrGyjzqi7wAeGldKiEi0tA5Rmlpk0xb5mO6f+7u70YDJPYD2gJXRbsLSG+llLWYFmaMK4g9dopGAVYXR8oxOyf2VynrpzCItWPahxMy4lWEsfCbEYYWiohIFTzjVqNjuy8iXA8qu0Y0kbXXf+K2Baa5e2Esrq+ZtU6JW8Xaa0wTgZbAFilxAJ/F4kg5dzKuSpmSU9RE/AHwIvBToATYizDGfQ93/y7LcUREGq31MCAizsx6ANsAX0dFzwObmtnesZj2wGHRPmJxzYFjY3HNgOOBke5eFBW/TEhWJyVOfTLwqbtPiZ6PJYzsTosrAN7O8n4y3+fk7jOAM7PGi4hIQi3d52RmzwIfAR8TxgRsBVwMrCbc4wQh6YwFRpjZ5YRuvKsJ14TuWFMl9/Fm9iRwdzS5whTgXMLMDifF4uaa2RDCJZ6l0fmPB/YlNjTd3YvN7FrCTbczgVFRzBnAhe6+Kst7zHmy1uhGqk2BmWoxiYhkV4uzko8DjiNMD9QCmA6MAW6Nerpw99Jo8oS7gKFAK0Ky2sfdpyeOdzrhxtmbCNMXTQAOcvePEnGDgELgItZe6jnO3V+IB7n7fWbmUf0uB6YBF7j70KxvMJcZIk4Frgd6xcqmAde6e/JmKxERiXGgtLR2kpO73w7cniGugNBiOaOauBXAJdFWVVwJIYHdlOHcwwj3XtVIpuRkZhcAfyQ0z24E5gA9gBOBh82sg7v/paaVEBFp8BzQkhmZZW05XQo85O7J7Ps3M3uIML5eyUlEpApaMiO7rEPJe7J2SoykxwitKBERqcr6HEvewGRtOX1CxbHtZbYkff4mERFZY50mdW10siani4AnzGw+8E93LzGzpoT1QS4HTlhfFRQRaTDUKsqs0uRkZtMp/1F2IHTtlZjZQsKUFU0JwwqfBHqvx3qKiGzYHLyWRus1BlW1nF5DeV5EpBYpOWVVaXJy94F5rIeISMOnr/uZ5TxDhIiI1JCSU2Y5JadorfmtCdNglOPuf6+tSomINDi6CTcnWWeI6Aj8G/hRWVH0GP8eoOQkIlIF3YSbXdabcG8hLMW+FyExHUWYZfZR4Btg9/VSOxGRhqTUsm2SOTkdSEhQ46LnM9x9jLufSphv76L1UTkRkYbEPNsm2ZPTxsA30Yy0K4F2sX3/BA6p7YqJiDQoWacuUnICsien2YQ1PgC+BfrH9vWrEC0iIgkWBkRk2STzaL3/EhLSi8AjwHVm1oew6uJplF/yV0RE0qhVlFnW5HQ9sEn0852EwRHHA60JienC2q+aiEgDU1rXFdhwZEpO7v418HX0czFhfadL12O9REQaFt3nlBPNECEikicaiZddVbOS/y6H47i731gL9RERabiUnDKrquU0OIfjOKDkJCIitaKqWcmzDjOXOjb549YcuMnOdV0NyUlRXVdA6oC69bLTNScRkXxwNDVRDpScRETyRS2nzJScRETyRN162Sk5iYjki5JTZkpOIiL5ouSUmZKTiEgeaDmM3OS6TPuOhAUHuwDD3H22mfUD5rj70vVRQRGRBkOj9TLLukx7S2AE8HPCSrgOvEBYSuMOYDJw1Xqqo4hIg6CWU3ZZb7S9GdgfOAXoQUhQZV4irJQrIiJV0WKDmWXt1jsR+K27P2ZmTRP7pgB9arVWIiINja455SRrcuoCfF7JviZAy9qpjohIA6bklFnWbr0plF+aPW53YFLtVEdEpOGy0mxbtccxO8bMnjGzb81shZlNMrNbzaxdIq6TmQ03s/lmtszMRpnZDinHa2Vmd5rZrOh4Y81sr5S4JmZ2tZlNNbOVZjbBzI6upI5nmdkXZlYU1e+c7J9U9uT0d+AqMzsJaBGVuZntA1wM/C2Xk4qIyDq5DCgBrgEOAu4FzgVeNbMmAGZmhJXKDyKsVn400BwYbWabJY73AHAW8DvgUGAW8IqZ7ZyIu5GwYsWfgYOBccBTZvazeJCZnQUMA56Jzv8UMNTMzs36BrN2690B7AQ8AgyPyv4LtAKecPc/ZT2hiEijVXvdeoe5+7zY8zfMrAB4GBgAvA4cDuwJ7OvuowHMbCyhJ+wK4NdR2U7AL4Az3P3BqOwNYCJwQ3QczKw7ISne5u53RecdHd1OdBvwnyiuGWEQ3SPuPigWtwlwo5kNj1ZUr1KmlpO7l7j7CcDewO8JCeqP0Zs+KcsxREQaNV97I251W7WHKp+YyrwfPW4aPR4OfFeWmKLXLSbcBnRE7HWHA8XAk7G41cATwIHRrUQQRmW3INxWFDcC2MHM+kbP+wPdUuIeIYxf2LO69wc53oTr7m8Bb+XyGhERiazfARF7R49lg9e2Az5NiZsInGpmbd29MIqb4u7LU+JaAP2in7cjLET2VUocwLaEVtl20fPkueNxo6mGFhQUEcmX7Pc5dTWzD2Lb2VUd1sw2JXTBjXL3D6LizsDClPCC6LFTxrjOscdF7p5MsWlxpBwzGVelrDNElFJNznf35P1PIiISMbKNxIvMd/ddMx3XrC3wL2A1cHrilGl/t5NzKK2POCqJzSxrt94NKSfqAhxAuMfpoXWphIhIg7cebsI1s1aEEXnfA/Z29xmx3QWkt1LKWkwLY3G9qogriD12MjNLtJ7S4ojOPSsW1zmxv0qZkpO7D04rj2aLeAFYnOU4IiKNWi0mJzNrThiqvTuwv7t/kgiZSGhAJG0LTIuuN5XFHWVmrRPXnbYFVrH2GtNEQmNkC8pfd9o2evwsFgfh2tOsKuKqtE7XnNy9BBgK/GZdjiMi0ijU0tx60b1MjwL7AUe4+7iUsOeBTc1s79jr2gOHRfvicc2BY2NxzYDjgZHuXhQVv0xIVskR2icDn7r7lOj5WGB+JXEFwNvVv8PaWc+pJRkvcImINGa12K33F0IyuRlYZmY/iu2bEXXvPU9IFCPM7HJCN97VhGtCd5QFu/t4M3sSuDtqjU0h3NDbl1iCcfe5ZjYEuNrMlgIfERLYvsSGprt7sZldS7jpdiYwKoo5A7jQ3VdleYNZB0Sk9Ue2ALYn3Hz1Qcp+ERGJq73kdHD0OCja4q4HBrt7qZkdCtxF6OFqRUhW+7j79MRrTickupuAjsAE4CB3/ygRNwgoBC4CehKmrjvO3V+IB7n7fWbmwKXA5cA04AJ3H5r1DWZtOU2l8lEaXwPnZz2hiEij5DmN1qv6UO59MsYVEFosZ1QTtwK4JNqqiishJLCbMpx7GGEKoxrJmpxOTylbCXwLvB9VWEREqqJZyTOrNjlFI/LGE6bBSJsyQ0REMtB6TtllGa3nhGtKu6znuoiINGxaCTezaltO0UW16UCbPNRHRKRhUuLJSdb7nIYBvzGzFtVGiohIBUbtzUreGGQdENGOcFfwN2b2MuGu3/hH6O5+XW1XTkSkIVHiya7S5GRm3wBHufsEwmqLZdKGJDqg5CQiUhUlp8yqajn1Icz+gLtraQ0RkXWl5JRZbUxfJCIi1dH1pJxUl5z0UYqI1Bb9Rc2suuR0vZnNz3Acd/fTaqNCIiINVW1NX9QYVJecdiasGV8dfR8QEamGuvWyqy45Henu7+WlJiIiDZluws2JBkSIiOSLklNmSk4iInlQNkOEZKPkJCKSJ1aq7JRVpclJN96KiNQiXXPKiVpOIiJ5om697JScRETyRckpMyUnEZE8UcspOyUnEZF8UXLKTMlJRCQfXNMX5ULJSUQkD3SfU26UnKRe2m3fJRx/wVz67bCC0lKY+U1Lht+0MRPeblfXVWu07nj6K3b68bLUfR+Mbsegk74HQO+tVnLqFbP5/g+W0aZ9KXOmN+eVJzrz7PBulJZYPqtc/7iyU1aNOjmZ2UDgwUp2/9TdR+WxOhWY2RgAdx9Ql/XIt5+dvIDzb57B8w925bG7e2DmbLH9SlptpP/YdenPV29G63Yl5cq+/8PlnHP9d4wd2R6Azj2KueOZr1gwuzn3Xbcpiwuassuehfzy2ll07LqaB27epC6qXm+o5ZRdo05OMccCMxJln9VFRRq7Hput4pzrZzL8xk14dni3NeUfvtG+DmslANO+bFWh7OCTClhVZLzxr44A7LH/Ejp2KeGSI7Zk5jctAZjwdjs27rOK/Y5Z2LiTk27CzYmSUzDe3b+qLsjMWrp7liVEpIYOPLEAd+PFR7rUdVWkGi1alfKTQxfx7qvtWboo/Clp3iL89V2+tPwEM8sWN6WJ5pzRgIgc6J9LJcxsoJm5me1lZk+Z2SLg3Wjfbmb2tJnNMLMVZjbJzG4xs40Sx5hqZg+lHNvNbHCi7AQz+8LMisxsopkdtf7eXf213W7LmP5VSwYcsYgH3/mc/0ybwINvf85hA7OseSn59H8HL6ZNu1JefarTmrI3X+jAogVNOf/mmfTYvIjWbUv48UGL2e+YhTwzrFsVR2scrDTbJmo5lWlqZvHPIt74fhR4HDiGtZ9XL2A88BCwFNgO+B3wPeCEXE9uZvsDjwH/Bi4FugH3AM2BSbkeb0PWpWcxnXsU88trv+OhWzfmu29bsNehi7nglpk0beo894D+wNUX+x9bwMJ5zXj/9bVdrovmN+fiw7Zk8INT+Pu7XwBQWgojft+Dp4Z2r6uq1g+OBkTkQMkp+CLx/G1gePTz0+5+RXynuz9T9rOZWRS/BPi7mZ3v7gtyPP/1UR2OcPfS6LifA+OoJDmZ2dnA2QCtaJ3j6eovM6dNu1JuOLM3b7/UEQjXLHpsvorjL5zLcw90JQzKlbrUuUcxu/ykkOeGdy03Aq9D59VcO3wqK5c34cZf9mbJwmbsvGchJ140l+JVTfjHXxp3gtKAiOyUnIKjKD8gYinQP/r52WSwmbUHBhFaU5sTWjhltgQyJyczawrsBtxWlpgA3P1dM5ta2evc/X7gfoD21rnB/JNfsrAZsIqP3iw/ZPzDN9qx275L6dxjNQVzmqe/WPJmv6MX0rQpjHqqc7nyY8+fS4/NV3Hq7t+ncHH48/Lx2LY0aeKcevlsXn68M0sKGvGfnQbzP3X9a8T/Ssr5NDkgwszKktOslPgHgf0JXXnjgWXA7sBfgIpDmqrWlZDc5qTsSytr0L6d3Iptd11eodyiL+eu/vh6Yf9jFvL1xFZ881m5y6z03WYl301tuSYxlZk0vjXNWzib9ClqtMlJN+HmRgMiqlfun5OZtQKOAO5093vc/Q13/wBYkfLalUCLxOs7J2LmA8VAj5TXp5U1aO+81AGAHw5YWq78hwOWMO+75iycp1ZTXdtyx+X02WYlr/4j+U8ZCuY2Y5M+RbTtsLpc+Ta7hC8cC2Y34t+fO1aabRMlp5poCTQlJJS4gSmx3wLbJ8oOjT9x9xLgfeAYM1vz+zCzPYA+61jXDc57r7Vj/H/bctHtMzjklPn8YO+lXHTHdHYdUMjDd/Ss6+oJsP+xC1ldDKOf7Vhh378f6ULLVqXc8vg37HXYInbecymnXj6Lo8+Zx3//04F537WoeMDGxDNuGZjZZmb2JzMba2bLo1HAfVLiOpnZcDObb2bLzGyUme2QEtfKzO40s1nRKOSxZrZXSlwTM7s6Go280swmmNnRldTxrNgo5Elmdk62d6duvZy5+2IzGwdcamazCC2fM4BNU8KfAP5mZkOAF4GdSE9i1wEjgefMbBhhtN71wOzafwf1nXH9GX04/ZpZnHLZHNp2KGH6Vy257fxejH62U/Uvl/WqaTNnwJEL+WBMexbNr9gK+uKjNlx6VD9OungO59wwkzbtwvRFjw7poaHk1Hq3Xj/gOOBD4C3ggArnCwO2ngf6AhcCC4GrgdFmtrO7x6+1PwAcAlwOfAOcD7xiZv3dfXws7kbgMsJ19w8JI5SfMrND3f0/sXOfBQwDbgVGAfsBQ83M3P3e6t6ceSMe2hibvmjLlGtOVe3rA9wL7EnozvsH8BIhAe3j7mOiuCbAb4EzCQnnLeA84CvgencfHDvmicBgQmvpq+h1F0H10xe1t86+h+2XwzsXkVyN8qc/dPdda/r6dh028x/s+etMsW/+58pqz2VmTWKje38J/BXo6+5TYzFHAM8B+7r76KisAzAFGOHuv47KdiJcPz/D3R+MypoBE4FJ7n54VNYdmE4YwHVd7DyvAd3cfcfYa78DXnL302JxfwMOBzZ292TvUzmNuuXk7g8R7lXKdd9U4OCUXZaIKwVuiLZK46LYxwn3U8VVGCkoIhuwWmwLxEf3VuFw4LuyxBS9brGZvUC4dv7rWFwx8GQsbrWZPQFcFZsd50DCdfQRifOMIPQS9XX3KYTRzt1S4h4BTid8sR9NFXTNSUQkT8yzbbVoO+DTlPKJQC8zaxuLm+LuyaGyEwnJqF8srojQu5OMA9g2FkfKuZNxlWrULScRkXzKYSReVzP7IPb8/ujexlx1BqamlBdEj52AwihuYRVxnWOPi7zi9aC0OFKOmYyrlJKTiEg+5DYr+fx1ub4VY5WcNXlpYX3EUUlsJurWExHJg3ATrmfaalEB6a2UsqGvCzPGFcQeO0WjAKuLI+WYnRP7K6XkJCKSL6UZt9ozkbXXf+K2Baa5e2Esrq+ZJSfq3BZYxdprTBMJ93pukRIHa9fBK7u2lDx3Mq5SSk4iInlSBy2n54FNzWzvNXUIc4MeFu2LxzUnLLxaFtcMOB4YGVvH7mVCsjopcZ6TCdPATYmejyXcA5oWV0CYLLtKuuYkIpIP62ElXDM7Jvrxh9HjwWY2D5jn7m8Qks5YYISZXc7am3ANuGNN1dzHm9mTwN1m1pxwH9S5hJt3T4rFzY0mFbjazJYCHxES2L6EoellccVmdi3hptuZhJtw9yVMWHChu6+q7r0pOYmI5MV6mTfvqcTzodHjG8AAdy81s0OBu6J9rQjJah93n5547enAzcBNQEdgAnCQu3+UiBtEGOF3EdCTsKzPce7+QjzI3e8zMyesUXc5MA24wN2HkoGSk4hIvtTyjDzuXu3iZu5eQGixnFFN3ArgkmirKq6EkMBuynDuYYQpjHKm5CQikg+uJdhzoeQkIpIvjXgu01wpOYmI5ItyU2ZKTiIieWKl6tfLSslJRCQfnNq+wbZBU3ISEckDo9ZvsG3QlJxERPJFySkzJScRkXxRcspMyUlEJB90zSknSk4iInmi0XrZKTmJiOSFq1svB0pOIiL54Cg55UDJSUQkX9Srl5mSk4hInug+p+yUnERE8kXJKTMlJxGRfHCHEvXrZaXkJCKSL2o5ZabkJCKSL0pOmSk5iYjkgwOlSk5ZKTmJiOSFg+uaU1ZKTiIi+eBoQEQOlJxERPJF15wyU3ISEckXJafMlJxERPJCE7/mQslJRCQfHNCSGZkpOYmI5ItaTpkpOYmI5IWmL8qFkpOISD44uO5zykzJSUQkXzRDRGZKTiIi+aJrTpkpOYmI5IO7RuvlQMlJRCRf1HLKTMlJRCQvHC8pqetKbDCUnERE8kFLZuREyUlEJF80lDwzJScRkTxwwNVyykzJSUQkH1yLDeZCyUlEJE80ICI7cw1t3OCZ2Tzg27qux3rSFZhf15WQzBry76u3u3er6YvN7GXC55PFfHc/qKbnagiUnKReM7MP3H3Xuq6HZKPfl9SWJnVdARERkSQlJxERqXeUnKS+u7+uKyA50e9LaoWuOYmISL2jlpOIiNQ7Sk4iIlLvKDkJZjbQzNzMFplZp8S+ZtG+wXVQr/5m9q6ZLYvqsHO+65Cmrj6P2hb7vadt+9eD+o0xszF1XQ+pG5ohQuI6AFcCV9V1RSIPACuAw4DlwOS6rU6DdSwwI1H2WV1URKSMkpPEjQQuNLO73X12XVbEzJoAWwM3u/vr1cS2dPei/NSsQRrv7l9VF6TPWfJJ3XoSd1P0OKi6QDPb3cxGmVlh1O32mpntnoh5yMxmmNkuZvaWmS03sy/N7Jxqjj0QKCH8+7w26maamjhmfzN7x8xWAHdE+04ws9fNbF5Ur/+Z2WmJY/eJjjcwUT4gKh8QK2tqZjeZ2ayo7mPMbLvqPpuGINblt5eZPWVmi4B3o327mdnT0e9hhZlNMrNbzGyjxDGmmtlDKceu0C0a/e6+MLMiM5toZketv3cnGwIlJ4mbBfwZONvMelcWZGY7Am8AnYCBwKlAe+ANM9spEd4eeAwYARwBvA/ca2b7VFGPfwN7Rj8/APQH4n+sOgBPAI8DB0fHB/ge8DRwEnAk8AIwvLpkWIXBwDXAo9HxRgLP1/BY9VnT6Npi2dY0tu9RYApwDGu7e3sB44FzgIOAe4AzgAdrcvLo+tZjwJfAz4E7o2NuXZPjScOgbj1Juh34FXAd4Q9Omt8BRcB+7r4IwMxeBaZGr/t5LLYdcJ67j47i3gQOAE4ERqcd3N3nmdnC6OkMdx+XCGkLnOzu/0q87payn6NuwTHAxsC5wH2VveE00cCQi4H73f2yqHikmZUAt+VyrA3AF4nnbwPDo5+fdvcr4jvd/Zmyn83MovglwN/N7Hx3X5Dj+a+P6nCEe1hTwsw+B8YBk3I8ljQQajlJOe5eAPweONXMKvvmuhfwYlliil63hNCq2DsRu7wsMUVxRYRvyL3WoZqrgReThWa2pZk9bmYzgeJo+yU1+wa+A9AG+Eei/IkaHKu+OwrYLbadGdv3bDLYzNqb2e1m9jXhS0ox8AhgwJa5nDhqpe1GSIJrFjty93cJX3akkVJykjRDgALghkr2dyZ0ASbNJnT1xS1MiSsCWtW4djDX3cstjGNmbYFXgZ0I3U8/IfzR+xvQsgbn2Dh6nJMoTz5vCD519w9iW7y1kvZ7fpDQpfdH4KeEz/n8aF+uv9euQHPSP9eG+FlLRurWkwrcvdDMbiW0oO5MCSkAeqaU94z2rW9pc271B3oDP3H3/5YVmlny3/jK6LFForxL4nnZH+UewMRYeY/cqrrBK/dZm1krwrXDwe5+T6x8h5TXriTxOZtZ50TMfELLK+1z7UHDXadMqqGWk1RmKDCTtSP44t4ADjGzdmUF0c+HRfvqQuvosbisILpudEQibg6h5bZ9ovyQxPOPgWXAcYnyE9atmhu8lkBTYp9zZGBK7LdU/JwPjT+JWsDvA8dE1wkBMLM9gD7rWFfZgKnlJKncvcjMbiB9lukbCX9kXjOz2wnfrq8kJIjKugLXt3cIF+X/YmbXEa4X/ZbwzbxDWZC7u5k9CZxpZpMJF9wPAQbED+bui8xsCDDIzJYSRuolr8c0Ou6+2MzGAZea2SzC53sGsGlK+BPA36LP8UVCl+vAlLjrCJ/vc2Y2DOhGGCRRp/faSd1Sy0mq8iBh8EI57v4x4Y/5EuBhwsXwQmBvd5+QzwrG6jSPcGG/KWE4+a2EEWcjUsIvAv5JGCr+JOE6yYUpcYOBW4BTCIM9DiC0Dhu7E4EPgb8ADxGSyEUpcQ+zdvTmC8CBlL8lAAB3H0UY/r814fdyOfAbNFKvUdOSGSIiUu+o5SQiIvWOkpOIiNQ7Sk4iIlLvKDmJiEi9o+QkIiL1jpKTiIjUO0pOUm+kLBu+1MwmmNkFKdMQ1fa5K6zzFK0dNTXH4wwws8Hx2Q5qqX6Dzaza+z4qW0Mp6/Fr63OO/S771MbxpPFRcpL66FjCXHlHA+8BfyIs05FvN5Jy02g1BhBuPNX/LZF1oOmLpD6KLxs+0sz6EWYMSE1QZtYcWO21fEe5u39dm8cTkez07U42BO8D7cyse6z77Twzu8PMviNM5NoRwMx+bmbjomXVF0VLjJdbO8rMWpvZUDNbYGE59+eBzZInTevWM7M2ZnabmX0dLSk+28yeMbMe0dLj10WhxWXdk4nz3m5mU8xsVfQ4KNkFaGuXtV9pZjPN7FrCWkk5M7NuZjbMzCZHn8l0M3vMzNLmwgP4vpmNjmJnmdkNKfXramb3RnUrsrC8+tk1qZ9IZdRykg1BX6CEMH9f2ezjgwhJ62zCfHorLSzHfi9hTsAbCKvwDiYsH7+juy+NXjsMOJ4wuej7hDWJypZ6r5SZtSCsGbUzYe6+cYRJZQ8krGM1nJDkziQsM18Se20z4BVgW0J34SfAj4BrCetjXRrFdQVeJ8xXdxoh8V5OzRdn7ExYuuJqYB6wSXSut81sG3dfmYh/jrAG1q3R+7oWKCV8jphZe8LKtxtFZVOiuHvNrKW7/6mG9RQpz921aasXG2HGaidMANqM8Af/V4Q/8s9FMX2imI+I5oaMytsCi4G/JY7ZB1gF/CZ6vnV0vKsScfdGxx0YK3sImBp7fkYUc3gV72FwFNMsUX5KVL5XonxQVL/u0fObo+e9YjFtCLN/e4bPcCrwUBX7mwKbR3U5KqXeyc/lr8BSoGP0/FpCstsyJW5+2fuO/S771PW/K20b5qZuPamPviCsF1RAWFfqUUJiiHvO3ePXmPoD7YFHzaxZ2QbMiI63VxS3B6E7uybLrx8AzHb353N5M5GDCOsbvZOo30jCSrA/ir2Pce4+reyF7r6MMKt3jZjZudGox0LCEvdlx05bvj7tc2nL2nWZDgLeBaYk3scrhAUbt61pPUXi1K0n9dFRhKSyFPjWK3Y9QcXlw7tHj6MqOWbZcvHrsvx6F8ICjDXRnbBSb3KRvvixIdTv05T9NVqy3MwuJCyn/gdC9+BCQnIeR/qS6pV9LmXXqLoD/aj+fYisEyUnqY8+9bWj9SqTHJm3IHocSPll1cuUXW+KL7/+TWx/luXX51NxZdesFhCuzyRX1i0zNXqcVUldaro8/AnAa+5+aVmBmfWtIr6yz6UsKS8A5pK+fhNoDSapJUpO0lC8Q0hA/dz94Sri3iVc4D8OuC1WnmX59ZHACWZ2mLtX1s1WFD1uxNqECPAy4b6tQnf/oopzjAUuN7PN3X06hBGC1HyRw9aERSHjTq8iPu1zKWRta+5lwsKM09x9bg3rJFItJSdpENx9iZldTlimvRvwEmGAxKbA3sAYd3/M3SeZ2WNA2RDpstF6P8twmhHAWcDjZnYrIdG1I4xWuztKOp9FsZea2UtAibt/QLhudjphafvfAxOAFsAWwOHAke6+HBgCnEe4v2swa0frrajhR/MycKWZXUO4oXlf4Jgq4s+KfS4HAr8EBrv7omj/EMJIx7ei5dcnEQZsbAP8xN2PqGE9RcpRcpIGw92Hmdl0wh/zXxAGGswE3gTGx0J/RWgNXEZIEK9H8f+t5vjFZnYA4V6ms6PHBYSh1QVR2IuEQRznEW4aNsKowmIzOxC4KnptX2AZ8DXwb8IIPdx9vpntB9xDWOZ8AXAf4f9qTWbJuIFwD9jFhGtMbxCSzjeVxB9BmJHjWkJyv4kw9L3sM1hsZj+O6nIlIfkvIiSpZ2pQP5FUWqZdRETqHQ0lFxGRekfJSURE6h0lJxERqXeUnEREpN5RchIRkXpHyUlEROodJScREal3lJxERKTe+X9aaT78qe5HGwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(pipe_lr_balanced, X_valid_fold, y_valid_fold, display_labels=['Non fraud', 'Fraud']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Important note: changing the class weight will **generally reduce accuracy**.\n",
    "\n",
    "- The original model was trying to maximize accuracy.\n",
    "- Now you're telling it to do something different.\n",
    "- But that can be fine, accuracy isn't the only metric that matters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9991738438978045"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr.score(X_valid_fold, y_valid_fold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9781481710969288"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_balanced.score(X_valid_fold, y_valid_fold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- So what's going on here?\n",
    "- Because there are so many more negative examples than positive, false positives affect accuracy much more than false negatives.\n",
    "- Thus, precision matters a lot more than recall.\n",
    "- So, the default method trades off a lot of recall for a bit of precision.\n",
    "- This maximizes accuracy overall - but do we like it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why do I have imbalance?\n",
    "\n",
    "- A very important question to ask yourself: \"Why do I have a class imbalance?\"\n",
    "  - Is it because one class is much more rare than the other?\n",
    "  - Is it because of my data collection methods?\n",
    "- If it's the data collection, then that means _your deployment and training data come from different distributions_!\n",
    "  - It's reasonable to use `class_weight` to try and fix this.\n",
    "  - If it's just because one is more rare than the other, you need to ask whether you care about one type of error more than the other.\n",
    "  - If so, it's again reasonable to use `class_weight` to address this.\n",
    "  \n",
    "But, if you answer \"no\" to both question, it may be fine to just ignore the class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `class_weight` vs. changing your threshold\n",
    "\n",
    "- Note that if you set `class_weight` because you care about one type of error more than the other, the output of `predict_proba` loses some of its meaning.\n",
    "- It's not really true that you have a 90% chance of fraud for this example. \n",
    "- It's really 20%, but you don't want false negatives.\n",
    "- What we're really doing is changing the threshold, but it's common to do it this way.\n",
    "  - From a conceptual standpoint, I think it's more useful to think about changing the threshold.\n",
    "- On the other hand, if you are using `class_weight` because it was harder to collect data for one of the classes, that is all fine.\n",
    "- Be careful about how you interpret the probabilities in general, and extra careful when using `class_weight`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stratified Splits (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A similar idea of \"balancing\" classes can be applied to data splits.\n",
    "- For example, with cross-validation, there is also [`StratifiedKFold`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html).\n",
    "- From the documentation:\n",
    "\n",
    "> This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.\n",
    "\n",
    "- In other words, if we have 10% negative examples in total, then each fold will have 10% negative examples.\n",
    "- We have the same option in `train_test_split` with the `stratify` argument. \n",
    "- Is this a good idea? \n",
    "  - Well, it's no longer a random sample, which is probably theoretically bad, but not that big of a deal.\n",
    "  - If you have many examples, it shouldn't matter as much.\n",
    "  - It can be especially useful in multi-class, say if you have one class with very few cases.\n",
    "  - In general, these are difficult questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_fold.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_valid_fold.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split into a certain ratio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(y_valid_fold)/len(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the number of frauds may not follow this exactly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(y_valid_fold==1).sum()/(y_train==1).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In this case, for a bigger dataset, it doesn't matter so much, but for small datasets it may.\n",
    "- I don't have strong feelings about this, we won't use it much in this course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring and hyperparameter tuning (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When you use `GridSearchCV` or `RandomizedSearchCV` there is an option to enter a custom scoring function.\n",
    "  - E.g. `scoring='roc_auc'`\n",
    "- It's important to ask ourselves what scoring function we want to optimize.\n",
    "- Optimizing recall might not make sense because you'll just try to predict fraud for everything.\n",
    "- Something like F1 score, average precision score, or ROC AUC try to balance precision vs. recall.\n",
    "  - But make sure you know whether your score is looking at `predict` or `predict_proba`.\n",
    "- Speaking of which, what happens when you call `fit`? \n",
    "  - Answer: it optimizes some score that depends on the classifier, but usually related to accuracy.\n",
    "  - However, `class_weight` can influence this.\n",
    "- ROC AUC is generally considered a good score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But note that the ROC curve and associated AUC score aren't affected so strongly by `class_weight`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9885166695730907"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_valid_fold, pipe_lr.predict_proba(X_valid_fold)[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9831859677849994"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_valid_fold, pipe_lr_balanced.predict_proba(X_valid_fold)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Setting `class_weight` will shift all the probabilities toward one direction (e.g. Fraud).\n",
    "- But it won't necessarily help you _separate_ the classes for a given threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, it's really up to you to decide things like:\n",
    "\n",
    "- Do I plan to use the output of `predict_proba` in my application?\n",
    "- What types of errors do I care about?\n",
    "- What sort of threshold/`class_weight` do I want?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary (5 min)\n",
    "\n",
    "- Problem: \"surely 99.9% accuracy is good?!\"\n",
    "- Solution: look at metrics beyond accuracy (e.g. confusion matrix)\n",
    "- Problem: \"I care more about false negatives than false positives\"\n",
    "- Solution: change probability threshold or use `class_weight`\n",
    "- Problem: \"I have a lot more negative examples in my dataset\"\n",
    "- Solution: ask yourself, is that really a problem? If so, use `class_weight` or something like SMOTE\n",
    "\n",
    "So...\n",
    "\n",
    "- In our standard workflow we should be asking ourselves these important questions above.\n",
    "- We should keep an eye out for class imbalance.\n",
    "- Don't always rely on `score` just because it's a default in sklearn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluation metrics True/False (Piazza)\n",
    "\n",
    "1. In medical diagnosis, false positives are more damaging than false negatives (assume \"positive\" means the person has a disease, \"negative\" means they don't).\n",
    "2. In spam classification, false positives are more damaging than false negatives (assume \"positive\" means the email is spam, \"negative\" means they it's not).\n",
    "3. In the medical diagnosis, high recall is more important than high precision.\n",
    "4. If method A gets a higher accuracy than method B, that means its precision is also higher.\n",
    "5. If method A gets a higher accuracy than method B, that means its recall is also higher.\n",
    "\n",
    "<br><br><br><br><br><br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Do not read until after the True/False**\n",
    "\n",
    "Method A - higher accuracy but lower precision\n",
    "\n",
    "| neg  | pos |\n",
    "|-----|------|\n",
    "| 90  |  5     |\n",
    "| 5  |   0    |\n",
    "\n",
    "Method B - lower accuracy but higher precision\n",
    "\n",
    "| neg | pos  |\n",
    "|-----|------|\n",
    "|  80  |  15     |\n",
    "|  0  |  5    |\n",
    "\n",
    "- Method A accuracy: 90/100\n",
    "- Method B accuracy: 85/100\n",
    "- Method A precision: 0\n",
    "- Method B precision: 0.25\n",
    "\n",
    "Note (and my previous mistake): the total true positives and total true negatives need to be the same for both methods, otherwise this whole thing doesn't make sense!\n",
    "\n",
    "-------------\n",
    "\n",
    "Method A - higher accuracy but lower recall\n",
    "\n",
    "| neg | pos  |\n",
    "|-----|------|\n",
    "| 99| 0     |\n",
    "|  1  |  0    |\n",
    "\n",
    "Method B - lower accuracy but higher recall\n",
    "\n",
    "| neg | pos  |\n",
    "|-----|------|\n",
    "| 0  | 99     |\n",
    "|  0  |  1    |\n",
    "\n",
    "- Method A accuracy: 99/100\n",
    "- Method B accuracy: 1/100\n",
    "- Method A recall: 0\n",
    "- Method B recall: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
